@@ -113,7 +113,7 @@ need to retrieve a job using `ma_resource_manager_next_job()` and then process i
void my_custom_job_thread(...)
{
for (;;) {
ma_job job;
ma_resource_manager_job job;
ma_result result = ma_resource_manager_next_job(pMyResourceManager, &job);
if (result != MA_SUCCESS) {
if (result == MA_NOT_DATA_AVAILABLE) {
...
...
@@ -121,7 +121,7 @@ need to retrieve a job using `ma_resource_manager_next_job()` and then process i
// with MA_RESOURCE_MANAGER_FLAG_NON_BLOCKING.
continue;
} else if (result == MA_CANCELLED) {
// MA_JOB_QUIT was posted. Exit.
// MA_RESOURCE_MANAGER_JOB_QUIT was posted. Exit.
break;
} else {
// Some other error occurred.
...
...
@@ -134,7 +134,7 @@ need to retrieve a job using `ma_resource_manager_next_job()` and then process i
}
```
In the example above, the `MA_JOB_QUIT` event is the used as the termination indicator, but you can
In the example above, the `MA_RESOURCE_MANAGER_JOB_QUIT` event is the used as the termination indicator, but you can
use whatever you would like to terminate the thread. The call to `ma_resource_manager_next_job()`
is blocking by default, by can be configured to be non-blocking by initializing the resource
manager with the `MA_RESOURCE_MANAGER_FLAG_NON_BLOCKING` configuration flag.
...
...
@@ -243,9 +243,10 @@ Resources are managed in two main ways:
A resource managed data source (`ma_resource_manager_data_source`) encapsulates a data buffer or
data stream, depending on whether or not the data source was initialized with the
`MA_DATA_SOURCE_FLAG_STREAM` flag. If so, it will make use of a `ma_resource_manager_data_stream`
object. Otherwise it will use a `ma_resource_manager_data_buffer` object. Both of these objects
are data sources which means they can be used with any `ma_data_source_*()` API.
`MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_STREAM` flag. If so, it will make use of a
`ma_resource_manager_data_stream` object. Otherwise it will use a `ma_resource_manager_data_buffer`
object. Both of these objects are data sources which means they can be used with any
`ma_data_source_*()` API.
Another major feature of the resource manager is the ability to asynchronously decode audio files.
This relieves the audio thread of time-consuming decoding which can negatively affect scalability
...
...
@@ -273,51 +274,52 @@ available starting from the current position.
Data Buffers
------------
When the `MA_DATA_SOURCE_FLAG_STREAM` flag is excluded at initialization time, the resource manager
will try to load the data into an in-memory data buffer. Before doing so, however, it will first
check if the specified file has already been loaded. If so, it will increment a reference counter
and just use the already loaded data. This saves both time and memory. A binary search tree (BST)
is used for storing data buffers as it has good balance between efficiency and simplicity. The key
of the BST is a 64-bit hash of the file path that was passed into
When the `MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_STREAM` flag is excluded at initialization time, the
resource manager will try to load the data into an in-memory data buffer. Before doing so, however,
it will first check if the specified file has already been loaded. If so, it will increment a
reference counter and just use the already loaded data. This saves both time and memory. A binary
search tree (BST) is used for storing data buffers as it has good balance between efficiency and
simplicity. The key of the BST is a 64-bit hash of the file path that was passed into
`ma_resource_manager_data_source_init()`. The advantage of using a hash is that it saves memory
over storing the entire path, has faster comparisons, and results in a mostly balanced BST due to
the random nature of the hash. The disadvantage is that file names are case-sensitive. If this is
an issue, you should normalize your file names to upper- or lower-case before initializing your
data sources.
When a sound file has not already been loaded and the `MA_DATA_SOURCE_ASYNC` is excluded, the file
will be decoded synchronously by the calling thread. There are two options for controlling how the
audio is stored in the data buffer - encoded or decoded. When the `MA_DATA_SOURCE_DECODE` option is
excluded, the raw file data will be stored in memory. Otherwise the sound will be decoded before
storing it in memory. Synchronous loading is a very simple and standard process of simply adding an
item to the BST, allocating a block of memory and then decoding (if `MA_DATA_SOURCE_DECODE` is
specified).
When the `MA_DATA_SOURCE_ASYNC` flag is specified, loading of the data buffer is done
asynchronously. In this case, a job is posted to the queue to start loading and then the function
immediately returns, setting an internal result code to `MA_BUSY`. This result code is returned
when the program calls `ma_resource_manager_data_source_result()`. When decoding has fully
When a sound file has not already been loaded and the `MMA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_ASYNC`
is excluded, the file will be decoded synchronously by the calling thread. There are two options
for controlling how the audio is stored in the data buffer - encoded or decoded. When the
`MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_DECODE` option is excluded, the raw file data will be stored
in memory. Otherwise the sound will be decoded before storing it in memory. Synchronous loading is
a very simple and standard process of simply adding an item to the BST, allocating a block of
memory and then decoding (if `MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_DECODE` is specified).
When the `MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_ASYNC` flag is specified, loading of the data buffer
is done asynchronously. In this case, a job is posted to the queue to start loading and then the
function immediately returns, setting an internal result code to `MA_BUSY`. This result code is
returned when the program calls `ma_resource_manager_data_source_result()`. When decoding has fully
completed `MA_RESULT` will be returned. This can be used to know if loading has fully completed.
When loading asynchronously, a single job is posted to the queue of the type
`MA_JOB_LOAD_DATA_BUFFER_NODE`. This involves making a copy of the file path and associating it
with job. When the job is processed by the job thread, it will first load the file using the VFS
associated with the resource manager. When using a custom VFS, it's important that it be completely
thread-safe because it will be used from one or more job threads at the same time. Individual files
should only ever be accessed by one thread at a time, however. After opening the file via the VFS,
the job will determine whether or not the file is being decoded. If not, it simply allocates a
block of memory and loads the raw file contents into it and returns. On the other hand, when the
file is being decoded, it will first allocate a decoder on the heap and initialize it. Then it will
check if the length of the file is known. If so it will allocate a block of memory to store the
decoded output and initialize it to silence. If the size is unknown, it will allocate room for one
page. After memory has been allocated, the first page will be decoded. If the sound is shorter than
a page, the result code will be set to `MA_SUCCESS` and the completion event will be signalled and
loading is now complete. If, however, there is more to decode, a job with the code
`MA_JOB_PAGE_DATA_BUFFER_NODE` is posted. This job will decode the next page and perform the same
process if it reaches the end. If there is more to decode, the job will post another
`MA_JOB_PAGE_DATA_BUFFER_NODE` job which will keep on happening until the sound has been fully
decoded. For sounds of an unknown length, the buffer will be dynamically expanded as necessary,
and then shrunk with a final realloc() when the end of the file has been reached.
`MA_RESOURCE_MANAGER_JOB_LOAD_DATA_BUFFER_NODE`. This involves making a copy of the file path and
associating it with job. When the job is processed by the job thread, it will first load the file
using the VFS associated with the resource manager. When using a custom VFS, it's important that it
be completely thread-safe because it will be used from one or more job threads at the same time.
Individual files should only ever be accessed by one thread at a time, however. After opening the
file via the VFS, the job will determine whether or not the file is being decoded. If not, it
simply allocates a block of memory and loads the raw file contents into it and returns. On the
other hand, when the file is being decoded, it will first allocate a decoder on the heap and
initialize it. Then it will check if the length of the file is known. If so it will allocate a
block of memory to store the decoded output and initialize it to silence. If the size is unknown,
it will allocate room for one page. After memory has been allocated, the first page will be
decoded. If the sound is shorter than a page, the result code will be set to `MA_SUCCESS` and the
completion event will be signalled and loading is now complete. If, however, there is more to
decode, a job with the code `MA_RESOURCE_MANAGER_JOB_PAGE_DATA_BUFFER_NODE` is posted. This job
will decode the next page and perform the same process if it reaches the end. If there is more to
decode, the job will post another `MA_RESOURCE_MANAGER_JOB_PAGE_DATA_BUFFER_NODE` job which will
keep on happening until the sound has been fully decoded. For sounds of an unknown length, the
buffer will be dynamically expanded as necessary, and then shrunk with a final realloc() when the
end of the file has been reached.
Data Streams
...
...
@@ -327,8 +329,8 @@ large sounds like music tracks in games that would consume too much memory if fu
memory. After every frame from a page has been read, a job will be posted to load the next page
which is done from the VFS.
For data streams, the `MA_DATA_SOURCE_FLAG_ASYNC` flag will determine whether or not
initialization of the data source waits until the two pages have been decoded. When unset,
For data streams, the `MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_ASYNC` flag will determine whether or
not initialization of the data source waits until the two pages have been decoded. When unset,
`ma_resource_manager_data_source_init()` will wait until the two pages have been loaded, otherwise
it will return immediately.
...
...
@@ -1067,10 +1069,10 @@ Resource Manager Data Source Flags
==================================
The flags below are used for controlling how the resource manager should handle the loading and caching of data sources.
*/
#define MA_DATA_SOURCE_FLAG_STREAM 0x00000001 /* When set, does not load the entire data source in memory. Disk I/O will happen on job threads. */
#define MA_DATA_SOURCE_FLAG_DECODE 0x00000002 /* Decode data before storing in memory. When set, decoding is done at the resource manager level rather than the mixing thread. Results in faster mixing, but higher memory usage. */
#define MA_DATA_SOURCE_FLAG_ASYNC 0x00000004 /* When set, the resource manager will load the data source asynchronously. */
#define MA_DATA_SOURCE_FLAG_WAIT_INIT 0x00000008 /* When set, waits for initialization of the underlying data source before returning from ma_resource_manager_data_source_init(). */
#define MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_STREAM 0x00000001 /* When set, does not load the entire data source in memory. Disk I/O will happen on job threads. */
#define MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_DECODE 0x00000002 /* Decode data before storing in memory. When set, decoding is done at the resource manager level rather than the mixing thread. Results in faster mixing, but higher memory usage. */
#define MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_ASYNC 0x00000004 /* When set, the resource manager will load the data source asynchronously. */
#define MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_WAIT_INIT 0x00000008 /* When set, waits for initialization of the underlying data source before returning from ma_resource_manager_data_source_init(). */
#define MA_RESOURCE_MANAGER_JOB_CUSTOM 0x00000100 /* Number your custom job codes as (MA_RESOURCE_MANAGER_JOB_CUSTOM + 0), (MA_RESOURCE_MANAGER_JOB_CUSTOM + 1), etc. */
/*
...
...
@@ -1239,7 +1241,7 @@ typedef struct
wchar_t*pFilePathW;
ma_bool32decode;/* When set to true, the data buffer will be decoded. Otherwise it'll be encoded and will use a decoder for the connector. */
ma_async_notification*pInitNotification;/* Signalled when the data buffer has been initialized and the format/channels/rate can be retrieved. */
ma_async_notification*pDoneNotification;/* Signalled when the data buffer has been fully decoded. Will be passed through to MA_JOB_PAGE_DATA_BUFFER_NODE when decoding. */
ma_async_notification*pDoneNotification;/* Signalled when the data buffer has been fully decoded. Will be passed through to MA_RESOURCE_MANAGER_JOB_PAGE_DATA_BUFFER_NODE when decoding. */
ma_fence*pInitFence;/* Released when initialization of the decoder is complete. */
ma_fence*pDoneFence;/* Released if initialization of the decoder fails. Passed through to PAGE_DATA_BUFFER_NODE untouched if init is successful. */
MA_APIma_resultma_resource_manager_job_queue_next(ma_resource_manager_job_queue*pQueue,ma_resource_manager_job*pJob);/* Returns MA_CANCELLED if the next job is a quit job. */
/* Maximum job thread count will be restricted to this, but this may be removed later and replaced with a heap allocation thereby removing any limitation. */
ma_resource_manager*pResourceManager;/* A pointer to the resource manager that owns this data stream. */
ma_uint32flags;/* The flags that were passed used to initialize the stream. */
ma_decoderdecoder;/* Used for filling pages with data. This is only ever accessed by the job thread. The public API should never touch this. */
ma_bool32isDecoderInitialized;/* Required for determining whether or not the decoder should be uninitialized in MA_JOB_FREE_DATA_STREAM. */
ma_uint64totalLengthInPCMFrames;/* This is calculated when first loaded by the MA_JOB_LOAD_DATA_STREAM. */
ma_bool32isDecoderInitialized;/* Required for determining whether or not the decoder should be uninitialized in MA_RESOURCE_MANAGER_JOB_FREE_DATA_STREAM. */
ma_uint64totalLengthInPCMFrames;/* This is calculated when first loaded by the MA_RESOURCE_MANAGER_JOB_LOAD_DATA_STREAM. */
ma_uint32relativeCursor;/* The playback cursor, relative to the current page. Only ever accessed by the public API. Never accessed by the job thread. */
ma_uint64absoluteCursor;/* The playback cursor, in absolute position starting from the start of the file. */
ma_uint32currentPageIndex;/* Toggles between 0 and 1. Index 0 is the first half of pPageData. Index 1 is the second half. Only ever accessed by the public API. Never accessed by the job thread. */
...
...
@@ -1480,7 +1482,7 @@ struct ma_resource_manager
ma_resource_manager_data_buffer_node*pRootDataBufferNode;/* The root buffer in the binary tree. */
ma_mutexdataBufferBSTLock;/* For synchronizing access to the data buffer binary tree. */
ma_threadjobThreads[MA_RESOURCE_MANAGER_MAX_JOB_THREAD_COUNT];/* The threads for executing jobs. */
ma_job_queuejobQueue;/* Lock-free multi-consumer, multi-producer job queue for managing jobs for asynchronous decoding and streaming. */
ma_resource_manager_job_queuejobQueue;/* Lock-free multi-consumer, multi-producer job queue for managing jobs for asynchronous decoding and streaming. */
ma_default_vfsdefaultVFS;/* Only used if a custom VFS is not specified. */
ma_loglog;/* Only used if no log was specified in the config. */
MA_APIma_resultma_resource_manager_process_next_job(ma_resource_manager*pResourceManager);/* Returns MA_CANCELLED if a MA_JOB_QUIT job is found. In non-blocking mode, returns MA_NO_DATA_AVAILABLE if no jobs are available. */
MA_APIma_resultma_resource_manager_process_next_job(ma_resource_manager*pResourceManager);/* Returns MA_CANCELLED if a MA_RESOURCE_MANAGER_JOB_QUIT job is found. In non-blocking mode, returns MA_NO_DATA_AVAILABLE if no jobs are available. */
#define MA_SOUND_FLAG_NO_DEFAULT_ATTACHMENT 0x00000010 /* Do not attach to the endpoint by default. Useful for when setting up nodes in a complex graph system. */
#define MA_SOUND_FLAG_NO_PITCH 0x00000020 /* Disable pitch shifting with ma_sound_set_pitch() and ma_sound_group_set_pitch(). This is an optimization. */
Lock free queue implementation based on the paper by Michael and Scott: Nonblocking Algorithms and Preemption-Safe Locking on Multiprogrammed Shared Memory Multiprocessors
/* We need to put the job into memory before we do anything. */
pQueue->jobs[ma_job_extract_slot(slot)]=*pJob;
pQueue->jobs[ma_job_extract_slot(slot)].toc.allocation=slot;/* This will overwrite the job code. */
pQueue->jobs[ma_job_extract_slot(slot)].toc.breakup.code=pJob->toc.breakup.code;/* The job code needs to be applied again because the line above overwrote it. */
pQueue->jobs[ma_job_extract_slot(slot)].next=MA_JOB_ID_NONE;/* Reset for safety. */
pQueue->jobs[ma_resource_manager_job_extract_slot(slot)].toc.allocation=slot;/* This will overwrite the job code. */
pQueue->jobs[ma_resource_manager_job_extract_slot(slot)].toc.breakup.code=pJob->toc.breakup.code;/* The job code needs to be applied again because the line above overwrote it. */
pQueue->jobs[ma_resource_manager_job_extract_slot(slot)].next=MA_RESOURCE_MANAGER_JOB_ID_NONE;/* Reset for safety. */
/* The job is stored in memory so now we need to add it to our linked list. We only ever add items to the end of the list. */
/* Failed to post job. Probably ran out of memory. */
ma_log_postf(ma_resource_manager_get_log(pResourceManager),MA_LOG_LEVEL_ERROR,"Failed to post MA_JOB_LOAD_DATA_BUFFER_NODE job. %s.\n",ma_result_description(result));
ma_log_postf(ma_resource_manager_get_log(pResourceManager),MA_LOG_LEVEL_ERROR,"Failed to post MA_RESOURCE_MANAGER_JOB_LOAD_DATA_BUFFER_NODE job. %s.\n",ma_result_description(result));
/*
Fences were acquired before posting the job, but since the job was not able to
ma_log_postf(ma_resource_manager_get_log(pResourceManager),MA_LOG_LEVEL_ERROR,"Failed to post MA_JOB_FREE_DATA_BUFFER_NODE job. %s.\n",ma_result_description(result));
ma_log_postf(ma_resource_manager_get_log(pResourceManager),MA_LOG_LEVEL_ERROR,"Failed to post MA_RESOURCE_MANAGER_JOB_FREE_DATA_BUFFER_NODE job. %s.\n",ma_result_description(result));
/* We failed to post the job. Most likely there isn't enough room in the queue's buffer. */
ma_log_postf(ma_resource_manager_get_log(pResourceManager),MA_LOG_LEVEL_ERROR,"Failed to post MA_JOB_LOAD_DATA_BUFFER job. %s.\n",ma_result_description(result));
ma_log_postf(ma_resource_manager_get_log(pResourceManager),MA_LOG_LEVEL_ERROR,"Failed to post MA_RESOURCE_MANAGER_JOB_LOAD_DATA_BUFFER job. %s.\n",ma_result_description(result));
The public API is not allowed to touch the internal decoder so we need to use a job to perform the seek. When seeking, the job thread will assume both pages
are invalid and any content contained within them will be discarded and replaced with newly decoded data.
/* The data buffer is not getting deleted, but we may be getting executed out of order. If so, we need to push the job back onto the queue and return. */
returnma_resource_manager_post_job(pResourceManager,pJob);/* Attempting to execute out of order. Probably interleaved with a MA_JOB_FREE_DATA_BUFFER job. */
returnma_resource_manager_post_job(pResourceManager,pJob);/* Attempting to execute out of order. Probably interleaved with a MA_RESOURCE_MANAGER_JOB_FREE_DATA_BUFFER job. */
ma_log_postf(ma_resource_manager_get_log(pResourceManager),MA_LOG_LEVEL_ERROR,"Failed to post MA_JOB_PAGE_DATA_BUFFER_NODE job. %d\n",ma_result_description(result));
ma_log_postf(ma_resource_manager_get_log(pResourceManager),MA_LOG_LEVEL_ERROR,"Failed to post MA_RESOURCE_MANAGER_JOB_PAGE_DATA_BUFFER_NODE job. %d\n",ma_result_description(result));
result back to MA_BUSY to make it clear that there's still more to load.
*/
if(result==MA_SUCCESS){
ma_jobnewJob;
ma_resource_manager_jobnewJob;
newJob=*pJob;/* Everything is the same as the input job, except the execution order. */
newJob.order=ma_resource_manager_data_buffer_node_next_execution_order(pJob->pageDataBufferNode.pDataBufferNode);/* We need a fresh execution order. */
returnma_resource_manager_post_job(pResourceManager,pJob);/* Attempting to execute out of order. Probably interleaved with a MA_JOB_FREE_DATA_BUFFER job. */
returnma_resource_manager_post_job(pResourceManager,pJob);/* Attempting to execute out of order. Probably interleaved with a MA_RESOURCE_MANAGER_JOB_FREE_DATA_BUFFER job. */
