Parse name value pairs in csv line using shell script

Question

I have input which looks like the following 2 lines:

TASK1,6,INITIAL,2013-01-15 19:20:40,PREPARING,2013-01-15 19:21:12,SCHEDULED,2013-01-15 19:21:13,TRANSLATING,2013-01-15 19:21:13,LOADING,2013-01-15 19:36:37,COMPLETE,2013-01-15 19:36:42
TASK2,5,INITIAL,2013-01-15 19:20:44,PREPARING,2013-01-15 19:21:13,SCHEDULED,2013-01-15 19:21:14,TRANSLATING,2013-01-15 19:36:37,TERMINAL,2013-01-15 20:28:10

I need to loop through a file with these lines and for each line calculate several time differences... i'm fine on the calculations and such, but i'm having a devil of a time trying to figure out how to parse this "variable length" string of name value pairs...

Basically the # after the Task# is the count of "statuses" followed by those statuses and their occuring time.

What i'd like to do is get one of the lines and end up with something like this having the values assigned to their respective variables. (using first line as example):

$TASK_ID=TASK1
$STATUS_COUNT=6
$INITIAL=2013-01-15 19:20:40
$PREPARING=2013-01-15 19:21:12
$SCHEDULED=2013-01-15 19:21:12
$TRANSLATING=2013-01-15 19:21:13
$LOADING=2013-01-15 19:36:37
$COMPLETE=2013-01-15 19:36:42
$TERMINAL=<NULL>

Compounding the problem is that if a task is submitted more than once it will simply append the next round of statuses to the first set meaning i could end up with an input line like:

TASK1,11,INITIAL,2013-01-15 19:20:40,PREPARING,2013-01-15 19:21:12,SCHEDULED,2013-01-15 19:21:13,TRANSLATING,2013-01-15 19:21:13,LOADING,2013-01-15 19:36:37,COMPLETE,2013-01-15 19:36:42,INITIAL,2013-01-15 20:20:40,PREPARING,2013-01-15 20:21:12,SCHEDULED,2013-01-15 20:21:13,TRANSLATING,2013-01-15 20:21:13,TERMINAL,2013-01-15 20:36:42

I this case i would want my output to be:

$TASK_ID=TASK1
$STATUS_COUNT=11
$INITIAL=2013-01-15 20:20:40
$PREPARING=2013-01-15 20:21:12
$SCHEDULED=2013-01-15 20:21:12
$TRANSLATING=2013-01-15 20:21:13
$LOADING=<NULL>
$COMPLETE=<NULL>
$TERMINAL=2013-01-15 20:36:42

I'm pretty stumped on this, can anyone help?

Thanks in advance

glenn jackman · Accepted Answer · 2013-01-23 00:04:00Z

3

#!/bin/bash

# Splitting on commas, read the task ID and status count followed by all of the statuses,
# which we'll parse later.
while IFS=, read -r TASK_ID STATUS_COUNT STATUSES; do
(
    # Subtly, but importantly, we put the loop body inside parentheses so each loop
    # iteration runs in a sub-shell. This ensures that the $LOADING, $COMPLETE, etc.
    # variables we set don't leak into future iterations.

    echo "TASK_ID      = $TASK_ID"
    echo "STATUS_COUNT = $STATUS_COUNT"

    # Convert the comma-separated string $STATUSES into an array using `read -a'.
    IFS=, read -ra STATUSES <<< "$STATUSES"

    # Assign the statuses to named variables. A side benefit of this is that only the
    # last value of each status type is used.
    for ((i = 0; i < ${#STATUSES[@]}; i += 2)); do
        declare "${STATUSES[$i]}=${STATUSES[$((i+1))]}"
    done

    # Print each of the statuses, or <NULL> if that stage wasn't listed.
    echo "INITIAL      = ${INITIAL:-<NULL>}"
    echo "PREPARING    = ${PREPARING:-<NULL>}"
    echo "SCHEDULED    = ${SCHEDULED:-<NULL>}"
    echo "TRANSLATING  = ${TRANSLATING:-<NULL>}"
    echo "LOADING      = ${LOADING:-<NULL>}"
    echo "COMPLETE     = ${COMPLETE:-<NULL>}"
    echo "TERMINAL     = ${TERMINAL:-<NULL>}"

    echo
)
done

Output:

$ ./tasks < tasks.txt
TASK_ID      = TASK1
STATUS_COUNT = 6
INITIAL      = 2013-01-15 19:20:40
PREPARING    = 2013-01-15 19:21:12
SCHEDULED    = 2013-01-15 19:21:13
TRANSLATING  = 2013-01-15 19:21:13
LOADING      = 2013-01-15 19:36:37
COMPLETE     = 2013-01-15 19:36:42
TERMINAL     = <NULL>

TASK_ID      = TASK2
STATUS_COUNT = 5
INITIAL      = 2013-01-15 19:20:44
PREPARING    = 2013-01-15 19:21:13
SCHEDULED    = 2013-01-15 19:21:14
TRANSLATING  = 2013-01-15 19:36:37
LOADING      = <NULL>
COMPLETE     = <NULL>
TERMINAL     = 2013-01-15 20:28:10

TASK_ID      = TASK1
STATUS_COUNT = 11
INITIAL      = 2013-01-15 20:20:40
PREPARING    = 2013-01-15 20:21:12
SCHEDULED    = 2013-01-15 20:21:13
TRANSLATING  = 2013-01-15 20:21:13
LOADING      = 2013-01-15 19:36:37
COMPLETE     = 2013-01-15 19:36:42
TERMINAL     = 2013-01-15 20:36:42

(Glenn Jackman adding edit based on new requirement)

events=(INITIAL PREPARING SCHEDULED TRANSLATING LOADING COMPLETE TERMINAL)

while IFS=, read -r TASK_ID STATUS_COUNT rest; do
    IFS=, read -ra STATUSES <<< "$rest"

    for (( i=0; i < ${#STATUSES[@]}; i+=2 )); do
        # if this this the initial event, reset all statuses
        if [[ ${STATUSES[i]} == ${events[0]} ]]; then
            for event in "${events[@]}"; do
                declare "$event="
            done
        fi
        declare "${STATUSES[i]}=${STATUSES[i+1]}"
    done
    for var in TASK_ID STATUS_COUNT "${events[@]}"; do
        printf "$%s = %s\n" $var "${!var:-<NULL>}"
    done

done

edited Jan 23, 2013 at 0:04

glenn jackman

249k42 gold badges233 silver badges362 bronze badges

answered Jan 22, 2013 at 23:08

John Kugelman

364k70 gold badges555 silver badges600 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

John Kugelman Over a year ago

@glennjackman I believe I recently learned about read -a from you, so thanks again for that gem!

user2001998 Over a year ago

Only issue here is that in the last case (where the task was run more than once)... i would want LOADING and COMPLETE to be NULL since those happened in the first run (set of statuses) but not in the second set... INITIAL always starts a new run if that helps break runs up.

user2001998 Over a year ago

using some date math i was able to make this solution work by adding some statements to NULL out timestamps < INITIAL... Thanks for your help

glenn jackman Over a year ago

@user2001998, I added an update to John's answer that handles this new requirement. John, hope you're OK with it.

Collectives™ on Stack Overflow

Parse name value pairs in csv line using shell script

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related