The exact details about thread-safety should be specified by the language. Currently, there is no such specification for Swift. There's even no such thing like a "thread". Thus, we should apply the "worst case" (e.g. how C would behave) and apply those knowledge from GCD, and other C APIs that can be used in Swift.
Notice, that some language idioms may be thread-safe only because the language and the underlying toolset itself takes care of it, e.g. inserting calls to appropriate memory-barriers instructions when required for the language construct and the current hardware when it generates the code for it. The language C (and C++) does nothing in this respect on our behalf.
What we strive to avoid is a "data race". A data race can happen when any thread writes to a memory location and any other thread reads from the same location without using explicit synchronisation primitives. And, think of a "thread" as the same thing we mean when creating a NSThread object in Cocoa.
So, your question whether your scenario is thread-safe is a clear "No, it is not." What you need is some form of concurrency control, e.g. using a dedicated serial dispatch queue where you execute the access (reading and writing) to the array, or use locks in conjunction with a mutex (aka a "critical section").