Containers
Classes for data containers. The aim of these data containers it to wrap the data with useful set of method to access and to manipulate the data, as well as load and store it.
These containers are inherited from standard Python containers (e.g. object, list, and dictionary) to allow them to be used together with other tools and libraries.
Basic containers
ObjectContainer
dcase_util.containers.ObjectContainer
Container class for object inherited from standard object class.
Examples how to inherit class from ObjectContainer:
1class Multiplier(dcase_util.containers.ObjectContainer):
2 def __init__(self, multiplier=None, **kwargs):
3 super(Multiplier, self).__init__(**kwargs)
4 self.multiplier = multiplier
5
6 def __str__(self):
7 ui = dcase_util.ui.FancyStringifier()
8 output = super(Multiplier, self).__str__()
9 output+= ui.data(field='multiplier', value=self.multiplier)
10 return output
11
12 def multiply(self, data):
13 return self.multiplier * data
14
15m = Multiplier(10)
16m.show()
17# Multiplier :: Class
18# multiplier : 10
19
20print(m.multiply(5))
21# 50
22
23# Save object
24m.save('test.cpickle')
25
26# Load object
27m2 = Multiplier().load('test.cpickle')
28
29print(m2.multiply(5))
30# 50
31
32m2.show()
33# Multiplier :: Class
34# filename : test.cpickle
35# multiplier : 10
|
Container class for object inherited from standard object class. |
|
Load file |
|
Save file |
|
Print container content |
|
Log container content |
DictContainer
dcase_util.containers.DictContainer
Dictionary container class inherited from standard dict class.
Usage examples:
1import dcase_util
2
3d = dcase_util.containers.DictContainer(
4 {
5 'test': {
6 'field1': 1,
7 'field2': 2,
8 },
9 'test2': 100
10 }
11)
12d.show()
13# DictContainer
14# test
15# field1 : 1
16# field2 : 2
17# test2 : 100
18
19print(d.get_path('test.field1'))
20# 1
21
22print(d.get_path(['test', 'field1']))
23# 1
24
25print(d.get_path('test2'))
26# 100
27
28d.set_path('test.field2', 200)
29print(d.get_path('test.field2'))
30# 200
31
32print(d.get_leaf_path_list())
33# ['test.field1', 'test.field2', 'test2']
34
35print(d.get_leaf_path_list(target_field_startswith='field'))
36# ['test.field1', 'test.field2']
37
38d.show()
39# DictContainer
40# test
41# field1 : 1
42# field2 : 200
43# test2 : 100
|
Dictionary container class inherited from standard dict class. |
|
Load file |
|
Save file |
|
Print container content |
|
Log container content |
|
Get value from nested dict with dotted path |
|
Set value in nested dict with dotted path |
Get path list to all leaf node in the nested dict. |
|
|
Recursive dict merge |
|
Get unique hash string for the data under given path. |
|
Get unique hash string (md5) for given parameter dict. |
ListContainer
dcase_util.containers.ListContainer
List container class inherited from standard list class.
|
List container class inherited from standard list class. |
|
Load file |
|
Save file |
|
Print container content |
|
Log container content |
|
Replace content with given list |
ListDictContainer
dcase_util.containers.ListDictContainer
List of dictionaries container class inherited from standard list class.
Usage examples:
1import dcase_util
2ld = dcase_util.containers.ListDictContainer(
3 [
4 {
5 'field1': 1,
6 'field2': 2,
7 },
8 {
9 'field1': 3,
10 'field2': 4,
11 },
12 ]
13)
14ld.show()
15# ListDictContainer
16# [0] =======================================
17# DictContainer
18# field1 : 1
19# field2 : 2
20#
21# [1] =======================================
22# DictContainer
23# field1 : 3
24# field2 : 4
25
26print(ld.search(key='field1', value=3))
27# DictContainer
28# field1 : 3
29# field2 : 4
30
31print(ld.get_field(field_name='field2'))
32# [2, 4]
|
List of dictionaries container class inherited from standard list class. |
|
Load file |
|
Save file |
|
Print container content |
|
Log container content |
|
Search in the list of dictionaries |
|
Get all data from field. |
RepositoryContainer
dcase_util.containers.RepositoryContainer
Container class for repository, inherited from dcase_util.containers.DictContainer
.
|
Container class for repository, inherited from DictContainer. |
|
Load file |
|
Save file |
|
Print container content |
|
Log container content |
TextContainer
dcase_util.containers.TextContainer
Container class for text, inherited from dcase_util.containers.ListContainer
.
|
Container class for text, inherited from ListContainer. |
|
Load file |
|
Save file |
|
Print container content |
|
Log container content |
Data containers
DataContainer
dcase_util.containers.DataContainer
Container class for data, inherited from dcase_util.containers.ObjectContainer
.
|
Container class for data, inherited from ObjectContainer. |
|
Load file |
|
Save file |
|
Print container content |
|
Log container content |
Data matrix |
|
Shape of data matrix |
|
Number of data columns |
|
Number of data frames |
|
Push processing chain item |
|
Focus segment start |
|
Focus segment stop |
|
Basic statistics of data matrix. |
|
Reset focus segment |
|
Get focus segment from data array. |
|
Freeze focus segment, copy segment to be container's data. |
|
|
Get frames from data array. |
|
Visualize data array. |
DataArrayContainer
dcase_util.containers.DataArrayContainer
Container class for data, inherited from dcase_util.containers.DataContainer
.
|
Array data container class, inherited from DataContainer. |
|
Load file |
|
Save file |
|
Print container content |
|
Log container content |
Data matrix |
|
Shape of data matrix |
|
Number of data columns |
|
Number of data frames |
|
Push processing chain item |
|
Focus segment start |
|
Focus segment stop |
|
Basic statistics of data matrix. |
|
Reset focus segment |
|
Get focus segment from data array. |
|
Freeze focus segment, copy segment to be container's data. |
|
|
Get frames from data array. |
|
Visualize data array. |
DataMatrix2DContainer
dcase_util.containers.DataMatrix2DContainer
DataMatrix2DContainer is data container for two-dimensional data matrix (numpy.ndarray).
Basic usage examples:
1# Initialize container with random matrix 10x100, and set time resolution to 20ms
2data_container = dcase_util.containers.DataMatrix2DContainer(
3 data=numpy.random.rand(10,100),
4 time_resolution=0.02
5)
6
7# When storing, e.g., acoustic features, time resolution corresponds to feature extraction frame hop length.
8
9# Access data matrix directly
10print(data_container.data.shape)
11# (10, 100)
12
13# Show container information
14data_container.show()
15# DataMatrix2DContainer :: Class
16# Data
17# data : matrix (10,100)
18# Dimensions
19# time_axis : 1
20# data_axis : 0
21# Timing information
22# time_resolution : 0.02 sec
23# Meta
24# stats : Calculated
25# metadata : -
26# processing_chain : -
27# Duration
28# Frames : 100
29# Seconds : 2.00 sec
The container has focus mechanism to flexibly capture only part of the data matrix. Focusing can be done based on time (in seconds, if time resolution is defined), or based on frame ids.
Examples using focus mechanism, accessing data and visualizing data:
1# Using focus to get part data between 0.5 sec and 1.0 sec
2print(data_container.set_focus(start_seconds=0.5, stop_seconds=1.0).get_focused().shape)
3# (10, 25)
4
5# Using focus to get part data between frame 10 and 50
6print(data_container.set_focus(start=10, stop=50).get_focused().shape)
7# (10, 40)
8
9# Resetting focus and accessing full data matrix
10data_container.reset_focus()
11print(data_container.get_focused().shape)
12# (10, 100)
13
14# Access frames 1, 2, 10, and 30
15data_container.get_frames(frame_ids=[1,2,10,30])
16
17# Access frames 1-5, and only first value per column
18data_container.get_frames(frame_ids=[1,2,3,4,5], vector_ids=[0])
19
20# Transpose matrix
21transposed_data = data_container.T
22print(transposed_data.shape)
23# (100, 10)
24
25# Plot data
26data_container.plot()
|
Two-dimensional data matrix container class, inherited from DataContainer. |
|
Load file |
|
Save file |
|
Print container content |
|
Log container content |
Data matrix |
|
Shape of data matrix |
|
Number of data columns |
|
Number of data frames |
|
Data vector length |
|
Push processing chain item |
|
Focus segment start |
|
Focus segment stop |
|
Transposed data in a data container |
|
Basic statistics of data matrix. |
|
Reset focus segment |
|
Get focus segment from data matrix. |
|
Freeze focus segment, copy segment to be container's data. |
|
Get frames from data matrix. |
|
|
Visualize data matrix. |
DataMatrix3DContainer
dcase_util.containers.DataMatrix3DContainer
|
Three-dimensional data matrix container class, inherited from DataMatrix2DContainer. |
|
Load file |
|
Save file |
|
Print container content |
|
Log container content |
Data matrix |
|
Number of data columns |
|
Number of data frames |
BinaryMatrix2DContainer
dcase_util.containers.BinaryMatrix2DContainer
|
Two-dimensional data matrix container class, inherited from DataContainer. |
|
Load file |
|
Save file |
|
Print container content |
|
Log container content |
Data matrix |
|
Number of data columns |
|
Number of data frames |
|
|
Pad binary matrix along time axis |
|
Visualize binary matrix, and optionally synced data matrix. |
DataRepository
dcase_util.containers.DataRepository
DataRepository is container which can be used to store multiple other data containers. Repository stores data with two level information: label and stream. The label is higher level key and stream is second level one. Repositories can be used, for example, to store multiple different acoustic features all related to same audio signal. Stream id can be used to store features extracted from different audio channels. Later features can be access using extractor label and stream id.
Usage examples:
1# Initialize container with data
2data_repository = dcase_util.containers.DataRepository(
3 data={
4 'label1': {
5 'stream0': {
6 'data': 100
7 },
8 'stream1': {
9 'data': 200
10 }
11 },
12 'label2': {
13 'stream0': {
14 'data': 300
15 },
16 'stream1': {
17 'data': 400
18 }
19 }
20 }
21)
22# Show container information::
23data_repository. show()
24# DataRepository :: Class
25# Repository info
26# Item class : DataMatrix2DContainer
27# Item count : 2
28# Labels : ['label1', 'label2']
29# Content
30# [label1][stream1] : {'data': 200}
31# [label1][stream0] : {'data': 100}
32# [label2][stream1] : {'data': 400}
33# [label2][stream0] : {'data': 300}
34
35# Accessing data inside repository
36data_repository.get_container(label='label1',stream_id='stream1')
37# {'data': 200}
38
39# Setting data
40data_repository.set_container(label='label3',stream_id='stream0', container={'data':500})
41data_repository. show()
42# DataRepository :: Class
43# Repository info
44# Item class : DataMatrix2DContainer
45# Item count : 3
46# Labels : ['label1', 'label2', 'label3']
47# Content
48# [label1][stream1] : {'data': 200}
49# [label1][stream0] : {'data': 100}
50# [label2][stream1] : {'data': 400}
51# [label2][stream0] : {'data': 300}
52# [label3][stream0] : {'data': 500}
|
Data repository container class to store multiple DataContainers together. |
|
Load file list |
|
Get container from repository |
|
Store container to repository |
Push processing chain item |
|
|
Visualize data stored in the repository. |
Audio containers
AudioContainer
dcase_util.containers.AudioContainer
AudioContainer is data container for multi-channel audio. It reads many formats (WAV, FLAC, M4A, WEBM) and writes WAV and FLAC files. Downloading audio content directly from Youtube is also supported.
Basic usage examples:
1# Generating two-channel audio
2audio_container = dcase_util.containers.AudioContainer(fs=44100)
3t = numpy.linspace(0, 2, 2 * audio_container.fs, endpoint=False)
4x1 = numpy.sin(220 * 2 * numpy.pi * t)
5x2 = numpy.sin(440 * 2 * numpy.pi * t)
6audio_container.data = numpy.vstack([x1, x2])
7
8audio_container.show()
9# AudioContainer :: Class
10# Sampling rate : 44100
11# Channels : 2
12# Duration
13# Seconds : 2.00 sec
14# Milliseconds : 2000.00 ms
15# Samples : 88200 samples
16
17# Loading audio file
18audio_container = dcase_util.containers.AudioContainer().load(
19 filename=dcase_util.utils.Example.audio_filename()
20)
21
22# Loading audio content from Youtube
23audio_container = dcase_util.containers.AudioContainer().load_from_youtube(
24 query_id='2ceUOv8A3FE',
25 start=1,
26 stop=5
27)
The container has focus mechanism to flexibly capture only part of the audio data while keeping full audio signal intact. Focusing can be done based on time (in seconds, if time resolution is defined), or based on sample ids. Focusing can be done to single channel or mixdown (mono) channels. Audio containers content can be replaced with focus segment by freezing it.
Examples using focus segment mechanism:
1# Using focus to get part data between 0.5 sec and 1.0 sec
2print(audio_container.set_focus(start_seconds=0.5, stop_seconds=1.0).get_focused().shape)
3# (2, 22050)
4
5# Using focus to get part data starting 5 sec with 2 sec duration
6print(audio_container.set_focus(start_seconds=5, duration_seconds=2.0).get_focused().shape)
7# (2, 88200)
8
9# Using focus to get part data starting 5 sec with 2 sec duration, mixdown of two stereo channels
10print(audio_container.set_focus(start_seconds=5, duration_seconds=2.0, channel='mixdown').get_focused().shape)
11# (88200,)
12
13# Using focus to get part data starting 5 sec with 2 sec duration, left of two stereo channels
14print(audio_container.set_focus(start_seconds=5, duration_seconds=2.0, channel='left').get_focused().shape)
15# (88200,)
16
17# Using focus to get part data starting 5 sec with 2 sec duration, seconds audio channel (indexing starts from 0)
18print(audio_container.set_focus(start_seconds=5, duration_seconds=2.0, channel=1).get_focused().shape)
19# (88200,)
20
21# Using focus to get part data between samples 44100 and 88200
22print(audio_container.set_focus(start=44100, stop=88200).get_focused().shape)
23# (2, 44100)
24
25# Resetting focus and accessing full data matrix::
26audio_container.reset_focus()
27print(audio_container.get_focused().shape)
28# (2, 441001)
29
30# Using focus to get part data starting 5 sec with 2 sec duration, and freeze this segment ::
31audio_container.set_focus(start_seconds=5, duration_seconds=2.0).freeze()
32print(audio_container.shape)
33# (2, 88200)
Processing examples:
1# Normalizing audio
2audio_container.normalize()
3
4# Resampling audio to target sampling rate
5audio_container.resample(target_fs=16000)
Visualizations examples:
1# Plotting waveform
2audio_container.plot_wave()
3
4# Plotting spectrogram
5audio_container.plot_spec()
(Source code, png, hires.png, pdf)
(Source code, png, hires.png, pdf)
|
Audio container class. |
|
Load file |
|
Load audio data from youtube |
|
Save audio |
|
Print container content |
|
Log container content |
Audio data |
|
Focus segment start in samples. |
|
Focus segment start in seconds. |
|
Focus segment stop in samples. |
|
Focus segment stop in seconds. |
|
Focus channel |
|
Audio load status. |
|
Audio data shape. |
|
Length of audio data in samples. |
|
Duration of audio data in samples. |
|
Duration of audio data in milliseconds. |
|
Duration of audio data in seconds. |
|
Rename channels for compatibility. |
|
Check if audio data is empty. |
|
Reset focus segment. |
|
|
Set focus segment |
Get focus segment from audio data. |
|
Freeze focus segment, copy segment to be container's data. |
|
|
Slice audio into overlapping frames. |
|
Normalize audio data. |
|
Resample audio data. |
Mix all audio channels into single channel. |
|
|
Simple sample overlay method |
|
Visualize audio data |
|
Visualize audio data as waveform. |
|
Visualize audio data as spectrogram. |
Feature containers
FeatureContainer
dcase_util.containers.FeatureContainer
|
Feature container class for a single feature matrix, inherited from DataContainer. |
FeatureRepository
dcase_util.containers.FeatureRepository
|
Feature repository container class to store multiple FeatureContainers together. |
Mapping containers
OneToOneMappingContainer
dcase_util.containers.OneToOneMappingContainer
|
Mapping container class for 1:1 data mapping, inherited from DictContainer class. |
|
Load file |
|
Save file |
|
Print container content |
|
Log container content |
|
Map with a key. |
Exchange map key and value pairs. |
Metadata containers
MetaDataItem
dcase_util.containers.MetaDataItem
|
Meta data item class, inherited from standard dict class. |
|
Print container content |
|
Log container content |
Unique item identifier |
|
Return item values in a list with specified order. |
|
Filename |
|
Filename |
|
Scene label |
|
Event label |
|
Onset |
|
Offset |
|
Identifier |
|
Source label |
|
Set label |
|
Tags |
|
|
Item active withing given segment. |
MetaDataContainer
dcase_util.containers.MetaDataContainer
|
Meta data container class, inherited from ListDictContainer. |
|
Log container content |
|
Log container content with all meta data items. |
|
Print container content |
|
Print container content with all meta data items. |
|
Load event list from delimited text file (csv-formatted) |
|
Save content to csv file |
|
Append item to the meta data list |
Number of files |
|
Number of events |
|
Number of unique scene labels |
|
Number of unique event labels |
|
Number of unique identifiers |
|
Number of unique tags |
|
Unique files |
|
Unique event labels |
|
Unique scene labels |
|
Unique tags |
|
Unique identifiers |
|
Unique source labels |
|
Find the offset (end-time) of last event |
|
|
Get container information in a string |
|
Filter content |
Filter time segment |
|
Process event content |
|
|
Map events with varying event labels into single target event label |
Get inactivity segments between events as event list |
|
Add time offset to event onset and offset timestamps |
|
|
Statistics of the container content |
Scene count statistics |
|
Event count statistics |
|
Tag count statistics |
|
Event roll |
|
|
Intersection of two meta containers |
Intersection report for two meta containers |
|
|
Difference of two meta containers |
Parameter containers
ParameterContainer
dcase_util.containers.ParameterContainer
|
Parameter container class for parameters, inherited from DictContainer class. |
AppParameterContainer
dcase_util.containers.AppParameterContainer
|
Parameter container class for application parameters, inherited from ParameterContainer. |
|
|
Process parameters |
|
|
Process parameter set |
|
Override container content recursively. |
Get data with path, path can contain string constants which will be translated. |
|
Set data with path, path can contain string constants which will be translated. |
|
Update active parameter set |
|
Get set ids |
|
Set id exists |
|
Get active set id |
|
|
Get parameter set |
DCASEAppParameterContainer
dcase_util.containers.DCASEAppParameterContainer
|
Parameter container class for DCASE application parameter files, inherited from AppParameterContainer. |
ParameterListContainer
dcase_util.containers.ParameterListContainer
|
Parameter list container, inherited from ListDictContainer. |
Probability containers
ProbabilityItem
dcase_util.containers.ProbabilityItem
|
Probability data item class, inherited from standard dict class. |
|
Print container content |
|
Log container content |
Filename |
|
Label |
|
|
|
Unique item identifier |
|
Return item values in a list with specified order. |
ProbabilityContainer
dcase_util.containers.ProbabilityContainer
|
Probability data container class, inherited from ListDictContainer. |
|
Print container content |
|
Log container content |
|
Load probability list from file |
|
Save content to csv file |
Append item to the meta data list |
|
Unique files |
|
Unique labels |
|
Unique indices |
|
|
Filter content |
|
Get probabilities as data matrix. |
Mixins
ContainerMixin
dcase_util.containers.ContainerMixin
|
Container mixin to give class basic container methods. |
|
Print container content |
|
Log container content |
FileMixin
dcase_util.containers.FileMixin
|
File mixin to give class methods to load and store content. |
Get file information, filename |
|
|
Detect file format from extension |
Validate file format |
|
Checks that file exists |
|
Check if file is empty |
|
|
Use csv.sniffer to guess delimiter for CSV file |
|
Determine if the file is compressed package. |
PackageMixin
dcase_util.containers.PackageMixin
|
Package mixin to give class basic methods to handle compressed file packages. |
Package password |
|
|
Extract the package. |