View Single Post
  #2  
Old December 14th, 2009, 01:49 PM posted to microsoft.public.access.tablesdbdesign
Jeff Boyce
external usenet poster
 
Posts: 1,555
Default "Normalizing" imported data

JOPO (just one person's opinion)...

Adding rows of data can slow down if your tables are indexed, as Access will
need to check existing rows to enforce constraints and do the indexing.

On the other hand, removing the index could mean loading bogus data.

Would your users be willing to trade a longer load time for guarantees that
they have good data?

--

Regards

Jeff Boyce
Microsoft Access MVP

Disclaimer: This author may have received products and services mentioned in
this post. Mention and/or description of a product or service herein does
not constitute endorsement thereof.

Any code or pseudocode included in this post is offered "as is", with no
guarantee as to suitability.

You can thank the FTC of the USA for making this disclaimer
possible/necessary.



"pccdavef" u56849@uwe wrote in message news:a093c0a1fd378@uwe...
Hello all -

I have a situation where I'm importing substantial amounts of data (30,000
to
120,000 rows at a pop) from three external sources through a text file or
an
Excel spreadsheet into three different data tables. I've established
lookup
tables for those fields that can reasonably be normalized between and
amongst
the data tables.

The process I'm going through is:

1. Import the raw data into an "import" table that matches the structure
of
the source data. Also included in the import table are columns for foreign
keys of 'normalizable' fields, which are set to 0 when the source data is
imported.

2. Append any new lookup data that may be present in the source file to
the
lookup tables.

3. Run a series of update queries on the import table to update the
foreign
key fields with the keys of the lookup data. Depending on the source data
file, there are between 3 and 7 of these update queries.

4. Append new records into the data table using only the foreign key
values
where applicable.

I'm discovering that the update queries in step 3 are taking a LONG time
to
run (several minutes each), which is going to annoy my users to no end.

My questions a
- are there other, better processes or data structures to use?
- is there a way of optimizing update queries?

I appreciate any help or suggestions -
Dave