46

Let's say I have this (MySQL) database, sorted by increasing timestamp:

Timestamp   System StatusA StatusB 
2011-01-01     A      Ok     Ok      
2011-01-02     B      Ok     Ok     
2011-01-03     A     Fail   Fail     
2011-01-04     B      Ok    Fail     
2011-01-05     A     Fail    Ok      
2011-01-06     A      Ok     Ok      
2011-01-07     B     Fail   Fail

How do I select the rows where StatusA changed from the previous row for that system? StatusB doesn't matter (I show it in this question only to illustrate that there may be many consecutive rows for each system where StatusA doesn't change). In the example above, the query should return the rows 2011-01-03 (StatusA changed between 2011-01-01 and 2011-01-03 for SystemA), 2011-01-06, 2011-01-07.

The query should execute quickly with the table having tens of thousands of records.

Thanks

Improve this question
2
  • Why would 2011-01-07 be returned and 2011-01-04 not? Commented Jul 2, 2011 at 23:32
  • 2
    @Andriy: 2011-01-04 is not returned because Status A is OK on both 2011-01-02 and 2011-01-04 (both for system B). Commented Jul 3, 2011 at 0:21

8 Answers 8

Reset to default
46 
SELECT a.*
FROM tableX AS a
WHERE a.StatusA <>
      ( SELECT b.StatusA
        FROM tableX AS b
        WHERE a.System = b.System
          AND a.Timestamp > b.Timestamp
        ORDER BY b.Timestamp DESC
        LIMIT 1
      )

But you can try this as well (with an index on (System,Timestamp):

SELECT System, Timestamp, StatusA, StatusB
FROM
  ( SELECT (@statusPre <> statusA AND @systemPre=System) AS statusChanged
         , System, Timestamp, StatusA, StatusB
         , @statusPre := StatusA
         , @systemPre := System
    FROM tableX
       , (SELECT @statusPre:=NULL, @systemPre:=NULL) AS d
    ORDER BY System
           , Timestamp
  ) AS good
WHERE statusChanged ;
Improve this answer
Sign up to request clarification or add additional context in comments.

11 Comments

The first query executes in about 13 seconds. There are a bit less than 5000 records in the database.
It can't be using indexes if 5K records aren't instantaneous. What do your indexes look like?
There's an index on System, StatusA, TimeStamp... What would you like to know when you ask "what the indexes look like". They're purple with pink polka dots :)
@ypercube: I don't know what those @, := do -- so I have to figure that out before I try the 2nd one :)
Sweet! The second query executed in 0.0172 secs!
|
12

Use rownum

I've got 0.05 seconds on 20000 rows

select a1.*
  from (select rownum R_NUM, TIMESTAMP, System, StatusA from TableX) a1 
  join (select rownum R_NUM, TIMESTAMP, SYSTEM, STATUSA from TABLEX) a2 
    on a1.R_NUM = a2.R_NUM+1 
 where a1.system = a2.system 
   and a1.StatusA != a2.StatusA
Improve this answer

2 Comments

The question is tagged mysql and rownum is only applicable to Oracle Dbs. Is there a mysql equivalent to this?
a1.R_NUM = a2.R_NUM+1 sounds like it would fail to capture the last row, and the case where there is exactly one row.
11 
select a.Timestamp, a.System, a.StatusA, a.StatusB
from tableX as a
cross join tableX as b
where a.System = b.System
and a.Timestamp > b.Timestamp
and not exists (select * 
    from tableX as c
    where a.System = c.System
    and a.Timestamp > c.Timestamp
    and c.Timestamp > b.Timestamp
)
and a.StatusA <> b.StatusA;

Update addressing a comment: Why not use an inner join instead of a cross join?

The question asks for a MySQL solution. According to the documentation:

In MySQL, CROSS JOIN is a syntactic equivalent to INNER JOIN (they can replace each other). In standard SQL, they are not equivalent. INNER JOIN is used with an ON clause, CROSS JOIN is used otherwise.

This means that either of these joins would work.

The conditional_expr used with ON is any conditional expression of the form that can be used in a WHERE clause. Generally, you should use the ON clause for conditions that specify how to join tables, and the WHERE clause to restrict which rows you want in the result set.

The condition a.System = b.System probably falls under the 'how to join tables' category so using an INNER JOIN would be nicer in this case.

Since both produce the same results, the difference might be in performance. To say which will be faster I would need to know how are the joins implemented internally - whether they use indexes or hashing to do the joining.

Improve this answer

6 Comments

A cross join B where A.x = B.x? Why not A inner join B on A.x = B.x? Otherwise, it's nice indeed! (+1)
I'm sure MySQL is smart enough to notice the similarity between CROSS JOIN + WHERE join_condition and INNER JOIN … ON join_condition. INNER JOIN just seemed more natural to me, thence my question. Otherwise I'm of the opinion that your original version is as correct and as efficient as the revised one. Thanks for taking the trouble of adding this extended explanation to your answer!
@Andriy: I think that syntactic equivalence means that CROSS JOIN ... WHERE join_condition will produce same plan as JOIN ... WHERE join_condition. And CROSS JOIN ... ON join_condition (not standard SQL) the same as JOIN ... ON join_condition. I agree that it will be smart enough to have all 4 versions producing same plan.
Hmm, I can't get this query to complete either; same result as le dorfier's suggestion. Entering it in phpmyadmin just results in a long wait, and eventually phpmyadmin returns to home screen. This is on a Host Gator account, which is usually quite quick.
I don't know why this answer is getting so many upvotes -- ypercube's answer (with the @ variables) is by far the fastest, at least according to my tests.
|
4

This is exactly what windowing functions as built for. LAG will give the exact answer:

create table t1 (ts date, sys char(1),stata varchar(10),statb varchar(10));

insert into t1 values
('2011-01-01','A',' Ok','Ok'),
('2011-01-02','B',' Ok','Ok'),
('2011-01-03','A','Fail','Fail'),
('2011-01-04','B',' Ok','Fail'),
('2011-01-05','A','Fail','Ok'),
('2011-01-06','A',' Ok','Ok'),
('2011-01-07','B','Fail','Fail');

select * from (
select ts,sys,stata,lag(stata) over(partition by sys order by ts asc) as prev from t1
) as subsel where stata!=prev
Improve this answer

1 Comment

Very good answer, you may want to include the case where there is no previous i.e. where stata != prev or prev is null
3 
SELECT   a.*
FROM    (select row_number() over (partition by System order by Timestamp asc) as aRow, Timestamp, System, StatusA, StatusB from tableX) as a
left join (select row_number() over (partition by System order by Timestamp asc) as bRow, Timestamp, System, StatusA, StatusB from tableX) as b on a.aRow = b.bRow + 1 and a.System = b.System 
where (a.StatusA != b.StatusA or b.StatusA is null)

It will return first row and rows where value is different.

Improve this answer

Comments

2

Here's a slightly shorter version with similar logic. I've tested this so often I'm sure it's efficient; primarily because it eliminates the correlated subquery (WHERE NOT EXISIS).

"c" is in there to make sure that b is directly below a - it says c (between them) can't be found (via the NULL test).

SELECT a.Timestamp, a.System, a.StatusA, a.StatusB
FROM tableX AS a
JOIN tableX AS b
    ON a.System = b.System
    AND a.Timestamp > b.Timestamp
LEFT JOIN tableX AS c
    ON a.System = b.System
    AND a.Timestamp > c.Timestamp
    AND b.Timestamp < c.Timestamp
WHERE c.System IS NULL
    AND a.StatusA <> b.StatusA;
Improve this answer

5 Comments

dorfier: Don't you mean: LEFT JOIN tableX AS c ON a.System = c.System AND a.Timestamp > c.Timestamp AND c.Timestamp > b.Timestamp ?
Hmm, I can't seem to get this query to complete -- entering it in phpmyadmin just results in a long wait, and eventually phpmyadmin returns to home screen. I modified the query as ypercube suggested.
@le dorfier: I like this approach: What indexes does it need to have good performance? I tried various in my machine without good results. When the rows get 2x, time goes about 4x (the same happens with my first query and with Jiri's).
@le dorfier: Another thought. Have you used it in MySQL before or only with other systems? Perhaps MySQL optimizer is not clever enough to find a good plan for this.
Nope, i'm referring to MySQL. If you have an index on timestamp only, it's going to index directly to the record with almost no chance of duplicate values. Highest possible cardinality - you don't need any other fields in the index at all.
2

Egor's answer worked for me in MSSQL with one small change. Had to replace the ROWNUM statements with:

select row_number () over (order by TIMESTAMP) as R_NUM, ...
Improve this answer

Comments

0 
Select * from table
Qualify lag(StatusA) is distinct from StatusA over (Partition by System order by Timestamp)
;

Since mysql don't have Qualify and is distinct from functions, we can use subquery like the below:

SELECT
    Timestamp, System, StatusA, StatusB 
FROM (
    SELECT
        *, lag(StatusA) OVER (Partition by System order by Timestamp) as prev 
    FROM
         table
) a
WHERE
    a.prev != StatusA OR a.prev is null
;
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.