You seek just a crumb, a nugget, a clue. With it you dice, you slice and reshape until you feel a new.
The 800-Line JSON Wall
So anyone who has worked with managing aws has ran aws ec2 describe-instances where the terminal explodes with 800 lines of nested JSON. Reservations containing Instances containing NetworkInterfaces containing SecurityGroups containing… you get it. Somewhere in that mess was the single piece of information we need: which instances were tagged “Production” and actually running.
I could scroll. I could grep. I could write a Python script, or use the built in query and filter. Or I could learn jq properly which will actually benefit you as that is something that is usefull outside of aws also.
You stop scrolling through output and start extracting exactly what you need. From “what’s a dot” to one-liners that save hours.
We’re focusing on AWS because that’s where JSON pain lives and is a perfect example to this issue: EC2 instances, security groups, deeply nested objects with optional fields that vanish when you need them most. No installation guides here. If you can run jq '.' and see pretty-printed JSON, you’re ready.
Learning to See: The Five Gestures
Every jq journey starts the same way. You pipe JSON into it, watch it format beautifully, and think “cool, now what?” The answer is five core gestures that work on any JSON, from AWS to GitHub to Kubernetes. Master these and you can navigate anything.
The Foundation
| # | Gesture | Command | Tiny Result | Why It Matters |
|---|---|---|---|---|
| 1 | Dot | jq '.' | (whole, pretty JSON) | Start every investigation by seeing the terrain |
| 2 | Key | jq '.Reservations' | [ { … }, … ] | Zero in; the same move works on GitHub, Kubernetes, Stripe |
| 3a | Index | jq '.Reservations[0]' | { … } | Peek into the first drawer without dumping everything |
| 3b | Iterate | jq '.Reservations[].Instances[]' | { … } | Stream each instance into your pipeline |
| 4 | Pipe | jq '.Reservations[].Instances[] | .InstanceId' | i-0123… | Chain filters like Unix—debug by cutting the pipe |
| 5 | Optional | jq '.PublicIpAddress?' | null | AWS likes to hide keys; this keeps scripts alive |
The Breakthrough: EC2 Queries That Actually Work
Once you know the gestures, real problems become solvable.
Here are the queries I wish I’d known. Each one solved a specific problem, usually at 2 AM when documentation wasn’t helping.
| Spell | Command | Sample Output | Quick Note |
|---|---|---|---|
| Alive & State | jq '.Reservations[].Instances[] | {id:.InstanceId,state:.State.Name}' | {"id":"...","state":"..."} | Add ? on State when inspecting zombies |
| Has “Environment” Tag | jq '.Reservations[].Instances[] | select(.Tags?[] | .Key=="Environment")' | {…} | AWS filter can’t do “key exists” |
| Production Only | jq '.Reservations[].Instances[] | select(.Tags?[] | .Key=="Environment" and .Value=="Production")' | {…} | Use ascii_downcase if teammates argue about case |
| Name contains WebServer | jq '.Reservations[].Instances[] | select(.Tags?[] | .Key=="Name" and (.Value | contains("WebServer")))' | {…} | Substring > prefix; AWS limited to “starts with” |
| Running + Production IDs | jq '.Reservations[].Instances[] | select(.State.Name=="running" and (.Tags?[] | .Key=="Environment" and .Value=="Production")) | .InstanceId' | i‑0123… | Parentheses save you from Boolean mishaps |
Security Group Archaeology: Finding the Open Door
This is where jq shines and earns its keep. You can hunt through security rules, filter by CIDR blocks, find exactly which group has that one dangerous rule. These queries saved me during more than one security audit.
| Hunt | Command | Sample Output | Field Note |
|---|---|---|---|
| Rolodex | jq '.SecurityGroups[] | {id:.GroupId,name:.GroupName}' | {"id":"sg‑0abc","name":"web"} | Names mutate; IDs don’t |
| SSH Wide‑Open | jq '.SecurityGroups[] | select(.IpPermissions[]? | .IpProtocol=="tcp" and .FromPort?==22 and .ToPort?==22) | {id:.GroupId,name:.GroupName}' | {"id":"sg‑0abc",…} | Adjust range to catch 0‑65535 slip‑ups |
| CIDR 192.168.1.0/24 | jq '.SecurityGroups[] | {id:.GroupId,rules:[ .IpPermissions[]? | select(.IpRanges[]?.CidrIp=="192.168.1.0/24") ]} | select(.rules!=[])' | {"id":"sg‑0def","rules":[…]} | AWS filter sees only exact strings; jq can regex later |
Shaping Output: When jq Becomes Your API
The real power isn’t just extracting data. It’s reshaping it into exactly what your next tool needs. A Slack bot wants a clean object. A spreadsheet needs CSV. An SSH loop needs raw strings with no quotes.
I learned this when building monitoring dashboards. AWS gives you everything, but dashboards need specific shapes. These transforms turned raw AWS data into dashboard-ready JSON in one command.
| Sculpt | Command | Output | Usage |
|---|---|---|---|
| Slack‑sized object | jq '.Reservations[].Instances[] | {InstanceId:.InstanceId,Name:(.Tags?[] | select(.Key=="Name") | .Value // "N/A"),Type:.InstanceType}' | {"InstanceId":"i‑0123…","Name":"Web","Type":"t3.micro"} | Hand to bots, dashboards, humans |
| Array of IDs | jq '[ .Reservations[].Instances[] | select(.State.Name=="running") | .InstanceId ]' | ["i‑0123…","i‑0456…"] | Feed into further jq or xargs |
| Raw strings | jq -r '.Reservations[].Instances[] | .InstanceId' | i‑0123… | Perfect for ssh loops |
| Instant CSV | jq -r '.Reservations[].Instances[] | [.InstanceId,.State.Name] | @csv' | "i‑0123…","running" | Drop into Sheets; charts appear |
The One-Liner That Ended the Search
AWS filters do the coarse work (tag and state filtering). Then jq does the fine work: extract ID, name, IP, and security groups into a clean object.
aws ec2 describe-instances \
--filters "Name=tag:Environment,Values=Production" "Name=instance-state-name,Values=running" | \
jq '.Reservations[].Instances[] \| {InstanceId:.InstanceId,Name:(.Tags?[] \| select(.Key=="Name") \| .Value // "N/A"),PrivateIp:.PrivateIpAddress,SGs:[ .SecurityGroups[]?.GroupId ]}'
This pattern works everywhere. Let the source API filter what it can. Then use jq for the logic it can’t handle: combining conditions, reshaping output, handling optional fields. You get speed from the API filter and precision from jq.
After months of daily jq usage, certain patterns emerged. Not rules, just habits that kept saving time.
Build incrementally. Don’t write the full query at once. Start with .Reservations[], see what comes out. Add .Instances[], check again. Add the filter, verify. jq is REPL-friendly. Use that.
Use -r early, @csv late. Raw strings (-r) are for piping to other commands. CSV and formatters are for final output to humans or spreadsheets. Know which phase you’re in.
Default everything with //. AWS JSON is full of optional fields that vanish at the worst time. (.Tags?[] | select(.Key=="Name") | .Value // "N/A") saves you from null errors and confused humans reading your output.
Save the good ones. That one-liner you use weekly? Put it in ~/bin/list-prod-instances with a verb name. Future you will thank current you.
The investment was one evening learning the five gestures. The return was every AWS CLI command becoming scriptable, precise, and fast. Not bad for a tool that’s just 29 characters: jq '.Reservations[].Instances[]'
The JSON is still 800 lines. But now you only see the 3 lines that matter.
Buy Me a Coffee