cdp_backend.database package

Submodules

cdp_backend.database.constants module

class cdp_backend.database.constants.EventMinutesItemDecision

Bases: object

FAILED = 'Failed'

PASSED = 'Passed'

class cdp_backend.database.constants.MatterStatusDecision

Bases: object

ADOPTED = 'Adopted'

IN_PROGRESS = 'In Progress'

REJECTED = 'Rejected'

class cdp_backend.database.constants.Order

Bases: object

ASCENDING = 'ASCENDING'

DESCENDING = 'DESCENDING'

class cdp_backend.database.constants.RoleTitle

Bases: object

ALTERNATE = 'Alternate'

CHAIR = 'Chair'

COUNCILMEMBER = 'Councilmember'

COUNCILPRESIDENT = 'Council President'

MEMBER = 'Member'

SECRETARY = 'Secretary'

SUPERVISOR = 'Supervisor'

VICEPRESIDENT = 'Vice President'

VICE_CHAIR = 'Vice Chair'

class cdp_backend.database.constants.VoteDecision

Bases: object

Abstain and Absent can mean many things. It depends on each municipality what each legally dictates.

See here: https://mrsc.org/Home/Stay-Informed/MRSC-Insight/April-2013/How-Are-Abstentions-Handled-When-Counting-Votes.aspx

You may see “Non-Voting” reported as (NV) from Legistar for example.

ABSENT_APPROVE = 'Absent (Approve)'

ABSENT_NON_VOTING = 'Absent (Non-Voting)'

ABSENT_REJECT = 'Absent (Reject)'

ABSTAIN_APPROVE = 'Abstain (Approve)'

ABSTAIN_NON_VOTING = 'Abstain (Non-Voting)'

ABSTAIN_REJECT = 'Abstain (Reject)'

APPROVE = 'Approve'

REJECT = 'Reject'

cdp_backend.database.functions module

cdp_backend.database.functions.create_body(body: Body, start_datetime: datetime) → Body

cdp_backend.database.functions.create_event(body_ref: Body, event_datetime: datetime, static_thumbnail_ref: File | None = None, hover_thumbnail_ref: File | None = None, agenda_uri: str | None = None, minutes_uri: str | None = None, external_source_id: str | None = None, credentials_file: str | None = None) → Event

cdp_backend.database.functions.create_event_minutes_item(event_minutes_item: EventMinutesItem, event_ref: Event, minutes_item_ref: MinutesItem, index: int) → EventMinutesItem

cdp_backend.database.functions.create_event_minutes_item_file(event_minutes_item_ref: EventMinutesItem, supporting_file: SupportingFile, credentials_file: str | None = None) → EventMinutesItemFile

cdp_backend.database.functions.create_file(uri: str, credentials_file: str | None = None) → File

cdp_backend.database.functions.create_matter(matter: Matter) → Matter

cdp_backend.database.functions.create_matter_file(matter_ref: Matter, supporting_file: SupportingFile, credentials_file: str | None = None) → MatterFile

cdp_backend.database.functions.create_matter_sponsor(matter_ref: Matter, person_ref: Person, external_source_id: str | None = None) → MatterSponsor

cdp_backend.database.functions.create_matter_status(matter_ref: Matter, status: str, update_datetime: datetime, event_minutes_item_ref: EventMinutesItem | None = None, external_source_id: str | None = None) → Matter

cdp_backend.database.functions.create_minimal_event_minutes_item(event_ref: Event, minutes_item_ref: MinutesItem, index: int) → EventMinutesItem

cdp_backend.database.functions.create_minimal_person(person: Person) → Person

cdp_backend.database.functions.create_minutes_item(minutes_item: MinutesItem, matter_ref: Matter | None = None) → MinutesItem

cdp_backend.database.functions.create_person(person: Person, picture_ref: File | None = None, credentials_file: str | None = None) → Person

cdp_backend.database.functions.create_role(role: Role, person_ref: Person, seat_ref: Seat, start_datetime: datetime, body_ref: Body | None = None) → Role

cdp_backend.database.functions.create_seat(seat: Seat, image_ref: File | None) → Seat

cdp_backend.database.functions.create_session(session: Session, session_video_hosted_url: str, session_content_hash: str, event_ref: Event, credentials_file: str | None = None) → Session

cdp_backend.database.functions.create_transcript(transcript_file_ref: File, session_ref: Session, transcript: Transcript) → Transcript

cdp_backend.database.functions.create_vote(matter_ref: Matter, event_ref: Event, event_minutes_item_ref: EventMinutesItem, person_ref: Person, decision: str, in_majority: bool | None, external_source_id: str | None = None) → Vote

cdp_backend.database.functions.generate_and_attach_doc_hash_as_id(db_model: Model) → Model

Generate a SHA256 hash to use as the document key for storage using the primary keys of the database model.

Parameters:

db_model: Model

The initialized database model.

Returns:

db_model: Model

The updated database model with the doc key set.

cdp_backend.database.functions.get_all_of_collection(db_model: Model, credentials_file: str, batch_size: int = 1000) → list[Model]

Get all documents in a collection as a single list but request in batches.

Parameters:

db_model: Model

The CDP database model to get all documents for.

credentials_file: str

Path to Google Service Account Credentials JSON file.

batch_size: int

How many documents to request at a single time. Default: 1000

Returns:

documents: List[Model]

All documents in the model’s collection.

cdp_backend.database.functions.upload_db_model(db_model: Model, credentials_file: str, transaction: Transaction | None = None, batch: WriteBatch | None = None) → Model

Upload or update an existing database model.

Parameters:

db_model: Model

The database model to upload.

credentials_file: str

Path to Google Service Account Credentials JSON file.

transaction: Optional[Transaction]

The transaction to write this model during.

batch: Optional[WriteBatch]

The write batch to use during uploading this model.

Returns:

db_model: Model

The uploaded, or updated, database model.

cdp_backend.database.models module

class cdp_backend.database.models.Body(*args, **kwargs)

Bases: Model

A meeting body. This can be full council, a subcommittee, or “off-council” matters such as election debates.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'body'

description = None

end_datetime = None

external_source_id = None

id = None

is_active = None

name = None

start_datetime = None

class cdp_backend.database.models.Event(*args, **kwargs)

Bases: Model

An event can be a normally scheduled meeting, a special event such as a press conference or election debate, and, can be upcoming or historical.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

agenda_uri = None

body_ref = None

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'event'

event_datetime = None

external_source_id = None

hover_thumbnail_ref = None

id = None

minutes_uri = None

set_validator_kwargs(kwargs: Dict) → None

static_thumbnail_ref = None

class cdp_backend.database.models.EventMinutesItem(*args, **kwargs)

Bases: Model

A reference tying a specific minutes item to a specific event.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'event_minutes_item'

decision = None

event_ref = None

external_source_id = None

id = None

index = None

minutes_item_ref = None

class cdp_backend.database.models.EventMinutesItemFile(*args, **kwargs)

Bases: Model

Supporting files for an event minutes item.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'event_minutes_item_file'

event_minutes_item_ref = None

external_source_id = None

id = None

name = None

set_validator_kwargs(kwargs: Dict) → None

uri = None

class cdp_backend.database.models.File(*args, **kwargs)

Bases: Model

A file from the CDP file store.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'file'

description = None

id = None

media_type = None

name = None

set_validator_kwargs(kwargs: Dict) → None

uri = None

class cdp_backend.database.models.IndexedEventGram(*args, **kwargs)

Bases: Model

An n-gram that has already been scored to create, when taken altogether, an event relevance index.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'indexed_event_gram'

context_span = None

datetime_weighted_value = None

event_ref = None

id = None

stemmed_gram = None

unstemmed_gram = None

value = None

class cdp_backend.database.models.Matter(*args, **kwargs)

Bases: Model

A matter is a specific legislative document. A bill, resolution, initiative, etc.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'matter'

external_source_id = None

id = None

matter_type = None

name = None

title = None

class cdp_backend.database.models.MatterFile(*args, **kwargs)

Bases: Model

A document related to a matter.

This file is usually not stored in CDP infrastructure but a remote source.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'matter_file'

external_source_id = None

id = None

matter_ref = None

name = None

set_validator_kwargs(kwargs: Dict) → None

uri = None

class cdp_backend.database.models.MatterSponsor(*args, **kwargs)

Bases: Model

A reference tying a specific person and a matter together.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'matter_sponsor'

external_source_id = None

id = None

matter_ref = None

person_ref = None

class cdp_backend.database.models.MatterStatus(*args, **kwargs)

Bases: Model

A matter status is the status of a matter at any given time. Useful for tracking the timelines of matters. I.E. Return me a timeline of matter x.

The same matter will have multiple matter statuses. 1. MatterStatus of submitted 2. MatterStatus of passed 3. MatterStatus of signed 4. etc.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'matter_status'

event_minutes_item_ref = None

external_source_id = None

id = None

matter_ref = None

status = None

update_datetime = None

class cdp_backend.database.models.MinutesItem(*args, **kwargs)

Bases: Model

An item referenced during a meeting. This can be a matter but it can be a presentation or budget file, etc.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'minutes_item'

description = None

external_source_id = None

id = None

matter_ref = None

name = None

class cdp_backend.database.models.Person(*args, **kwargs)

Bases: Model

Primarily the council members, this could technically include the mayor or city manager, or any other “normal” presenters and attendees of meetings.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'person'

email = None

external_source_id = None

static generate_router_string(name: str) → str

id = None

is_active = None

name = None

phone = None

picture_ref = None

router_string = None

set_validator_kwargs(kwargs: Dict) → None

static strip_accents(name: str) → str

website = None

class cdp_backend.database.models.Role(*args, **kwargs)

Bases: Model

A role is a person’s job for a period of time in the city council. A person can (and generally does) have many roles. For example: a person has two terms as city council member for district four then a term as city council member for a citywide seat. Roles can also be tied to committee chairs. For example: a council member spends a term on the transportation committee and then spends a term on the finance committee.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

body_ref = None

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'role'

end_datetime = None

external_source_id = None

id = None

person_ref = None

seat_ref = None

start_datetime = None

title = None

class cdp_backend.database.models.Seat(*args, **kwargs)

Bases: Model

An electable office on the City Council. I.E. “Position 9”.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'seat'

electoral_area = None

electoral_type = None

external_source_id = None

id = None

image_ref = None

name = None

class cdp_backend.database.models.Session(*args, **kwargs)

Bases: Model

A session is a working period for an event. For example, An event could have a morning and afternoon session.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

caption_uri = None

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'session'

event_ref = None

external_source_id = None

id = None

session_content_hash = None

session_datetime = None

session_index = None

set_validator_kwargs(kwargs: Dict) → None

video_end_time = None

video_start_time = None

video_uri = None

class cdp_backend.database.models.Transcript(*args, **kwargs)

Bases: Model

A transcript is a document per-session.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'transcript'

confidence = None

created = None

file_ref = None

generator = None

id = None

session_ref = None

class cdp_backend.database.models.Vote(*args, **kwargs)

Bases: Model

A reference tying a specific person and an event minutes item together.

classmethod Example() → Model

class Meta

Bases: object

ignore_none_field = False

collection: Manager = <fireo.managers.managers.Manager object>

collection_name = 'vote'

decision = None

event_minutes_item_ref = None

event_ref = None

external_source_id = None

id = None

in_majority = None

matter_ref = None

person_ref = None

cdp_backend.database.types module

class cdp_backend.database.types.IndexedField(fieldPath: str, order: str)

Bases: DataClassJsonMixin

fieldPath: str

order: str

class cdp_backend.database.types.IndexedFieldSet(fields: List[cdp_backend.database.types.IndexedField])

Bases: DataClassJsonMixin

fields: List[IndexedField]

cdp_backend.database.validators module

class cdp_backend.database.validators.UniquenessValidation(is_unique: bool, conflicting_models: list[Model])

Bases: object

An object containing uniqueness data of a database model object.

Parameters:

is_unique: bool

A boolean on whether the model is unique by primary key in the database collection.

conflicting_models: List[Model]

All existing models that share the same primary keys as the input model.

conflicting_models: list[Model]

is_unique: bool

cdp_backend.database.validators.create_constant_value_validator(constant_cls: type) → Callable[[str], bool]

Create a validator func that validates a value is one of the valid values.

Parameters:

constant_cls: Type

The constant class that contains the valid values.

Returns:

validator_func: Callable[[str], bool]

The validator func.

Notes

Will always allow None as a valid option. To remove None as a viable input, set the database model field required=True.

See: https://github.com/CouncilDataProject/cdp-backend/pull/164

cdp_backend.database.validators.email_is_valid(email: str | None) → bool

Validate that a valid email was provided.

None is a valid option.

Parameters:

email: Optional[str]

The email to validate.

Returns:

status: bool

The validation status.

cdp_backend.database.validators.is_secure_uri(url: str) → bool

cdp_backend.database.validators.resource_exists(uri: str | None, **kwargs: Any) → bool

Validate that the URI provided points to an existing file.

None is a valid option.

Parameters:

uri: Optional[str]

The URI to validate resource existance for.

kwargs: Any

Any extra arguments needed for resource retrieval. I.e. google_credentials_file.

Returns:

status: bool

The validation status.

cdp_backend.database.validators.router_string_is_valid(router_string: str | None) → bool

Validate that the provided router string contains only lowercase alphabetic characters and optionally include a hyphen.

None is a valid option.

Parameters:

router_string: Optional[str]

The router string to validate.

Returns:

status: bool

The validation status.

cdp_backend.database.validators.time_duration_is_valid(time_duration: str | None) → bool

Validate that the provided time duration string is acceptable to FFmpeg. The validator is unnecessarily limited to HH:MM:SS. The spec is a little more flexible.

None is a valid option.

Parameters:

time_duration: Optional[str]

The time duration to validate.

Returns:

status: bool

The validation status.

cdp_backend.database.validators.try_url(url: str, resolve_func: ~typing.Callable = <function resource_exists>) → str

Given a URL, return the URL with the protocol that exists (http or https) with a preference for https.

Parameters:

url: str

The target resource url.

resolve_func: func(url: str) -> bool

A function that takes in a str URL and determines whether it is reachable. Default is our “resource_exists” func

Returns:

resource_url: str

The url with the correct protocol based on where the resource exists. If does not exist, a LookupError is raised.

Module contents

Database package for cdp_backend.