I have an email script that reads in emails from a database and collects them into one huge object. Now these emails can be BCC, CC, TO type emails which must be passed to our email provider via an API in their respective type.
Their API allows batches up to 1000 emails which isn't the issue. The issue is that in each batch I have to have one TO address at least. In other words say by chance a batch has 1000 'BCC' type emails in it and not one 'TO' type email addresses. Those 1000 emails will not get sent. When you are dealing with 20,000 emails there are 20 batches created and there could very well be a batch that has all BCC or CC type emails with none of them being a TO type address.
So my question is how can I assure a TO address is in each batch during processing?
Some code along with the current output is below.
while (addresses.length > 0) {
// I do some processing here. The addresses above is a map of emails that
// Something like: [ { label: 'to', address: 'jay#someaddress.com' } ].
// Then the processing removes the individual labels from each email and slams
// them into one key value for each email type using a filter function. We also
// append some other things that must be separated as well like the recipient
// variables. The sent information looks like this to our email provider.
{
from: 'jay99#someaddress.com',
subject: 'fff',
text: 'fff',
to: ['jay+3#someaddress.com', 'jay+8#someadd.com'],
bcc: [],
* * // WHAT happens here if all the email addresses reside here and none above in the 'TO' label? The email will fail. The code must ensure at least one 'TO' type address in each batch of 1000.**
cc: [],
'recipient-variables': {
'jay+3#someaddress.com': {},
'jay+8#someadd.com': {}
}
}
}
My initial thought was to randomize it some how but wasn't sure how to do that. I think you almost have to do something to the initial array before it even gets into putting the emails into the right slots (IE type of emails). Suggestions? And thank you for taking the time to read my question. :)
Randomizing the combinations will not solve your problem either, since it does in no way guarantee that there is a 'to' recipient in all of your batches.
One way to solve this issue would be to create to buckets of recipients, one with all 'to' recipients, and another one with all others. This shouldn't be too hard, given that you have the complete list of addresses.
Then always take one 'to' address and 999 other addresses, package them into a batch and send them off. Repeat until either one of the buckets is empty. If the others bucket is empty first, only take the remaining 'to' recipients and send off the email until they are all gone as well.
If your 'to' bucket is empty first, you will have to come up with a different solution. Maybe make up dummy recipients?
Related
Gmail treats any email with dot (.) in it as the same as without dots. For example mail sent to hello#gmail.com or he.ll.o#gmail.com or he.llo#gmail.com will be sent to the same address. How will you store this in the database so that you avoid duplicate usage of emails with dots in it. For example if a user registers first with salman.khan#gmail.com , he should not be able to use any variation of the same email like sal.mankhan#gmail.com or salman.khan#gmail.com . (Assume the database have 10M of users).
You could remove .'s prior to saving them on the server, eg...
let emailone = "hello#gmail.com"
let emailtwo = "hel.lo#gmail.com"
const normaliseEmails = email => email.split('#')[0].replaceAll(".", "") + "#" + email.split('#')[1]
console.log(normaliseEmails(emailone))
console.log(normaliseEmails(emailtwo))
It's good you're thinking about this, as it can bite you later. Unless you are never going to use or reflect back the email address of a user, you'll likely want to keep the dots so you can show them. So what to do?
If this is important problem to solve, I'd add a second column to your database, a canonical email. This would be the email addresses without the dots. Then, when looking up email addresses, "canonize" the input by removing the periods, and then look it up using this column.
Your database may be able to do this dynamically for you, depending on its features. Be cognizant that this might not be performant as the database grows, but may be okay short term.
You may also want consider other "features" of email addresses, such as the ability to add optional stuff before the #, as in joe+duplicate#example.com, which gets routed to joe#example.com. This is likely fine, but you may want to guard against it as well.
I currently have a few issues with my Firestore querying technique. As per this stackoverflow post I made recently, Querying with two array with firestore security rules
The answer proposed to add the the "ids" into a object, with the key as the id, and the value simply being "true". I have completed this, and now my structure looks like so:
This leaves me with this query:
db.collection('Depots')
.where(`products.${productId}`, '==', true)
.where(`users.${userId}`, '==', true)
.where('created', '>', 1585998560500)
.orderBy('created', 'asc')
.get();
This query leaves me with throwing an error, asking to create an index:
The query requires an index. You can create it here: ...
However, this tries to index the specific object key, i.e. QXooVYGBIFWKo6C so products.QXooVYGBIFWKo6C. Which is certianly not what I want, as this query changes, and can have an infinite number of possibilities, which means I would have to create another index for each key entry in order to query it.
Is there any way to solve this issue? I am assuming it needs to index this query due to the different operators used in the query, so I was wondering if there were any workarounds to this issue.
Thank you very much in advance.
What you have here is a map field, for which indexes should usually be created automatically.
That indeed means that you'll have as many indexes as you have products, which means:
You are limited in how many products you can have, as there is a maximum of 40,000 index entries per document.
You pay more per document, as you pay for the storage of each index.
If these are not what you want, you'll have to switch back to your original model, with the query limitations you had there. There doesn't seem to be a solution that fits both of your requirements.
After our discussion in chat, this is the starting point I would suggest. Who knows what the end architecture would look like, but I think this or very close to this. You say that a user can exist in multiple depots at the same time and multiple depots can contain the same products, also at the same time. You also said that a depot can never have more than 40 users at a given time, so an array of 40 users would certainly not encroach on Firestore's document limit of 1,048,576 bytes.
[collection]
<documentId>
- field: value
[depots]
<UUID>
- depotId: string "depot456"
- productCount: num 5,000
<UUID>
- depotId: string "depot789"
- productCount: num 4,500
[products]
<UUID>
- productId: string "lotion123"
- depotId: string "depot456"
- users: [string] ["user10", "user27", "user33"]
<UUID>
- productId: string "lotion123"
- depotId: string "depot789"
- users: [string] ["user10", "user17", "user50"]
[users]
<userId>
- depots: [string] ["depot456", "depot999"]
<userId>
- depots: [string] ["depot333", "depot999"]
In NoSQL, storage is cheap and computation isn't so denormalize your data as much as you need to make your queries possible and efficient (fast and cheap).
To find all depots in a single query where user10 and lotion123 are both true, query the products collection where productId equals x and users array-contains y and collect the depotId values from those results. If you want to preserve the array-contains operation for something else, you'd have to denormalize your data further (replace the array for a single user). Or you could split this query into two separate queries.
With this model, when a user leaves a depot, get all products where users array-contains that user and remove that userId from the array. And when a user joins a depot, get all products where depotId equals x and append that userId to the array.
Watch this video, and others by Rick, to get a solid handle on NoSQL: https://www.youtube.com/watch?v=HaEPXoXVf2k
#danwillm If you are not sure about the number of users and products then your DB structure seems unfit for this situation because there are size and length limitations of the firestore document.
You should rather create a separate collection for products and users i.e normalize your data and have a reference for the user in the product collection.
User :
{
userId: documentId,
name: John,
...otherInfo
}
Product :
{
productId: documentId,
createdBy: userId,
createdOn:date,
productName:"exa",
...otherInfo
}
This way you there will be the size of the document would be limited, i.e try avoiding using maps/arrays in firestore if you are not sure about there size.
Also, in this case, the number of queries would be increased but you don't need many indexes in this case.
I want to generate a login id automatically. Logic is to take the user's last name and append a number(like 1, then 2 , then 3). system will also check if this user id exist in database. I have cached the database data in a local variable. If the user id is unique, it will be populated in a text box, otherwise it will append a new number. But This process may go for 10 times, if 10 user has same last name. Could anybody provide a simple solution for this. I want this to be done in JQuery.
Let's say John Doe tries to register an account. You need to:
Send 'Doe' to backend via ajax
Execute a query like this to find the greatest Doe* in DB:
SELECT user_id FROM users WHERE surname = 'Doe' ORDER BY added_ts DESC LIMIT 1
If a record is found, extract the number part out of the user id and send it back, if no record is found simply send 0 back
When ajax response arrives, increment the number you get and concatenate it to the surname.
I am assuming you have a record timestamp of some sort to use in the query (added_ts).
Edit: #Jaromanda X 8 pointed correctly that the query might return wrong results and it should be the surname field we need to check.
I have one class Messages with 3 principal fields:
id FromUser ToUser
I do have a query where the To = Value field and the From field is not repeated. I mean, get all FROMUSER who sent me a message.
Any Idea?
Thanks!
As #Fosco says, "group by" or "select distinct" are not supported yet in Parse.com.
Moreover keep in mind the restriction on the selection limit (max 1000 results for query) and timeout request call ( 3 seconds in the before save events, 7/10 seconds in the custom functions ). For the "count" selection, the restriction is the timeout request call.
I'm working on Parse.com too, and i've changed a lot the structure of my db model, often adding some inconsistent columns in several classes, keeping them carefully updated for each necessary query.
For cases like yours, i suggest to make a custom function, that keep in input two parameter ( we can say, "myLimit" and "myOffset" ) for the lazy loading, then select the slices, and programmatically try to filter the resulting array item list (with a simple search using for..loop, or using some utility of UnderscoreJS). Start with small slices ( eg: 200-300 records maximum for selection ) until the last selection returns zero results ( end reached). You could count all items before start all of this, but the timeout limitation could cause you problems. If this not works as expected try to make the same, client side.
You could also make a different approach, so creating another table/class, and for each new message, adding the FromUser in that table ONLY if it doesn't already exist, for that specified ToUser.
Hope it helps
Below is a snipet of code that I am having trouble with. The purpose is to check duplicate entries in the database and return "h" with a boolean if true or false. For testing purposes I am returning a true boolean for "h" but by the time the alert(duplicate_count); line gets executed the duplicate_count is still 0. Even though the alert for a +1 gets executed.
To me it seems like the function updateUserFields is taking longer to execute so it's taking longer to finish before getting to the alert.
Any ideas or suggestions? Thanks!
var duplicate_count = 0
for (var i = 0; i < skill_id.length; i++) {
function updateUserFields(h) {
if(h) {
duplicate_count++;
alert("count +1");
} else {
alert("none found");
}
}
var g = new cfc_mentoring_find_mentor();
g.setCallbackHandler(updateUserFields);
g.is_relationship_duplicate(resource_id, mentee_id, section_id[i], skill_id[i], active_ind,table);
};
alert(duplicate_count);
There is no reason whatsoever to use client-side JavaScript/jQuery to remove duplicates from your database. Security concerns aside (and there are a lot of those), there is a much easier way to make sure the entries in your database are unique: use SQL.
SQL is capable of expressing the requirement that there be no duplicates in a table column, and the database engine will enforce that for you, never letting you insert a duplicate entry in the first place. The syntax varies very slightly by database engine, but whenever you create the table you can specify that a column must be unique.
Let's use SQLite as our example database engine. The relevant part of your problem is right now probably expressed with tables something like this:
CREATE TABLE Person(
id INTEGER PRIMARY KEY ASC,
-- Other fields here
);
CREATE TABLE MentorRelationship(
id INTEGER PRIMARY KEY ASC,
mentorID INTEGER,
menteeID INTEGER,
FOREIGN KEY (mentorID) REFERENCES Person(id),
FOREIGN KEY (menteeID) REFERENCES Person(id)
);
However, you can make enforce uniqueness i.e. require that any (mentorID, menteeID) pair is unique, by changing the pair (mentorID, menteeID) to be the primary key. This works because you are only allowed one copy of each primary key. Then, the MentorRelationship table becomes
CREATE TABLE MentorRelationship(
mentorID INTEGER,
menteeID INTEGER,
PRIMARY KEY (mentorID, menteeID),
FOREIGN KEY (mentorID) REFERENCES Person(id),
FOREIGN KEY (menteeID) REFERENCES Person(id)
);
EDIT: As per the comment, alerting the user to duplicates but not actually removing them
This is still much better with SQL than with JavaScript. When you do this in JavaScript, you read one database row at a time, send it over the network, wait for it to come to your page, process it, throw it away, and then request the next one. With SQL, all the hard work is done by the database engine, and you don't lose time by transferring unnecessary data over the network. Using the first set of table definitions above, you could write
SELECT mentorID, menteeID
FROM MentorRelationship
GROUP BY mentorID, menteeID
HAVING COUNT(*) > 1;
which will return all the (mentorID, menteeID) pairs that occur more than once.
Once you have a query like this working on the server (and are also pulling out all the information you want to show to the user, which is presumably more than just a pair of IDs), you need to send this over the network to the user's web browser. Essentially, on the server side you map a URL to return this information in some convenient form (JSON, XML, etc.), and on the client side you read this information by contacting that URL with an AJAX call (see jQuery's website for some code examples), and then display that information to the user. No need to write in JavaScript what a database engine will execute orders of magnitude faster.
EDIT 2: As per the second comment, checking whether an item is already in the database
Almost everything I said in the first edit applies, except for two changes: the schema and the query. The schema should become the second of the two schemas I posted, since you don't want the database engine to allow duplicates. Also, the query should be simply
SELECT COUNT(*) > 0
FROM MentorRelationship
WHERE mentorID = #mentorID AND menteeID = #menteeID;
where #mentorID and #menteeID are the items that the user selected, and are inserted into the query by a query builder library and not by string concatenation. Then, the server will get a true value if the item is already in the database, and a false value otherwise. The server can send that back to the client via AJAX as before, and the client (that's your JavaScript page) can alert the user if the item is already in the database.