I have this Sql Server database table called productPrices:
shopId int
price decimal(18, 2)
dateFound datetime
I want to make an sql query formatted to use on a chart.js line chart. The line chart takes the following parameters:
An array of X Values - which in this context will be the date/day (from lowest to highest)
Each shopId contains an array of prices.
I have made a codepen version of how the data should be displayed in the line chart (using just data added manually)
https://codepen.io/nickbuus/pen/OJxEGqK
The problem is that if no price was added for the shop on a particular day then a data value for instance 0 still needs to be present since price arrays length has to match the X Values array.
How could I make a query that fills out 0 when the particular shopId doesn't have a value on that particular date?
What is the best way to format the returned data from the sql query when it should be used for the structure that chart.js line chart uses?
Ideally what you should be doing here is creating a Calendar Table. I'm not going to cover how you create a calendar table here, as a search in your favourite search engine of something like "Calendar Table SQL Server" will give you a huge wealth of resources and give some great explanations and their use cases.
Once you have a Calendar table, you need to CROSS JOIN that to your Shops table; which I also assume you have. Then you can simply LEFT JOIN to your Prices table.
So a parametrised query might look like this:
SELECT S.ShopID,
COUNT(P.Price) AS Prices,
C.CalendarDate AS Datefound
FROM dbo.Calendar C
CROSS JOIN dbo.Shops S
LEFT JOIN dbo.Prices P ON C.CalendarDate = P.DateFound --Though DateFound is a datetime, I assume it's time portion is 00:00:00.000
AND S.ShopId = P.ShopID
WHERE C.CalendarDate >= #StartDate
AND C.CalendarDate < DATEADD(DAY, 1, #EndDate)
GROUP BY S.ShopID,
C.CalendarDate;
If you can't, for some reason, create a Calendar table and you don't have (and can't create) a Shop table (I strongly suggest you do create one though) then you'll need to use an inline tally to create your Calendar, and DISTINCT to get the Shop IDs.
A parametrised query would look something like this:
WITH N AS(
SELECT N
FROM (VALUES(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL))N(N)),
Tally AS(
SELECT 0 AS I
UNION ALL
SELECT TOP(DATEDIFF(DAY, #StartDate, #EndDate))
ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS I
FROM N N1, N N2, N N3), --1,000 rows. Add more cross joins for more rows
Calendar AS(
SELECT DATEADD(DAY, T.I, #StartDate) AS CalendarDate
FROM Tally T),
Shops AS(
SELECT DISTINCT P.ShopID
FROM dbo.Prices P)
SELECT S.ShopID,
COUNT(P.Price) AS Prices,
C.CalendarDate AS Datefound
FROM dbo.Calendar C
CROSS JOIN dbo.Shops S
LEFT JOIN dbo.Prices P ON C.CalendarDate = P.DateFound --Though DateFound is a datetime, I assume it's time portion is 00:00:00.000
AND S.ShopId = P.ShopID
GROUP BY S.ShopID,
C.CalendarDate;
db<>fiddle
The A column is first name and the B column is last name in the spreadsheet I'm manipulating. In the other sheet it's the full name(but not necessarily just firstname + lastname)
HC is the spreadsheet I need to get the data from. This code will work if the person has a unique first name that is the same in both sheets : =VLOOKUP(""&A2&"",HC!$C$1:$I$400,7,False) but this isn't enough in my case. The *s known as wildcards check if this string is contained in another string regardless of it's position.
I tried =VLOOKUP(""&A5&" "&B5&"",HC!$C$1:$I$400,7,False) but that means the two strings have to occur one after the other. I need it to return if search_key1 is contained within the string regardless of accents(á or a) or case (A or a) and the same for search_key2 e.g. the string could be "Seárch_key1 kdslfj SEARCH_KEY2 akldfj" and this would return True and then give the 7th column.
function VLOOKUP2(search_key1,search_key2,range,index){
//is search_key1 contained in one of the cells in the range if so is search_key2 also contained in one of the cells in the range regardless of capitilization and diacritical marks
// return the column specified by the index parameter
}
I know you're asking for a custom function, but I think you could do this with two existing vlookup functions.
If your data on sheet HC looks like this:
Then on your lookup sheet, try this in cell C1 (deleting everything below):
=arrayformula({"Result";if(A2:A<>"",iferror(iferror(vlookup("*"&A2:A&"*"&B2:B&"*",{HC!C:I},7,0),vlookup("*"&B2:B&"*"&A2:A&"*",{HC!C:I},7,0)),),)})
HELP NOTES
The inital vlookup tries to match "*"&A2:A&"*"&B2:B&"*" in {HC!C:I}. {} is used to create an array.
If not, then the iferror tries another vlookup to match "*"&B2:B&"*"&A2:A&"*".
Another iferror deals with no match, removing #N/A.
Then the if(A2:A<>"" only tries the above formula if there is a corresponding value in the same row, in Col A.
The final array {} places "Result" at the top of the column, with the main formula below. After the heading text, ; forces a return in the array.
Within the vlookup, 0 denotes false. 1 would be true.
I'm having a bit of a strange problem. I'm trying to add a foreign key to one table that references another, but it is failing for some reason. With my limited knowledge of MySQL, the only thing that could possibly be suspect is that there is a foreign key on a different table referencing the one I am trying to reference.
I've done a SHOW CREATE TABLE query on both tables, sourcecodes_tags is the table with the foreign key, sourcecodes is the referenced table.
CREATE TABLE `sourcecodes` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`user_id` int(11) unsigned NOT NULL,
`language_id` int(11) unsigned NOT NULL,
`category_id` int(11) unsigned NOT NULL,
`title` varchar(40) CHARACTER SET utf8 NOT NULL,
`description` text CHARACTER SET utf8 NOT NULL,
`views` int(11) unsigned NOT NULL,
`downloads` int(11) unsigned NOT NULL,
`time_posted` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `user_id` (`user_id`),
KEY `language_id` (`language_id`),
KEY `category_id` (`category_id`),
CONSTRAINT `sourcecodes_ibfk_3` FOREIGN KEY (`language_id`) REFERENCES `languages` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `sourcecodes_ibfk_1` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `sourcecodes_ibfk_2` FOREIGN KEY (`category_id`) REFERENCES `categories` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1
CREATE TABLE `sourcecodes_tags` (
`sourcecode_id` int(11) unsigned NOT NULL,
`tag_id` int(11) unsigned NOT NULL,
KEY `sourcecode_id` (`sourcecode_id`),
KEY `tag_id` (`tag_id`),
CONSTRAINT `sourcecodes_tags_ibfk_1` FOREIGN KEY (`tag_id`) REFERENCES `tags` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1
This is the code that generates the error:
ALTER TABLE sourcecodes_tags ADD FOREIGN KEY (sourcecode_id) REFERENCES sourcecodes (id) ON DELETE CASCADE ON UPDATE CASCADE
Quite likely your sourcecodes_tags table contains sourcecode_id values that no longer exists in your sourcecodes table. You have to get rid of those first.
Here's a query that can find those IDs:
SELECT DISTINCT sourcecode_id FROM
sourcecodes_tags tags LEFT JOIN sourcecodes sc ON tags.sourcecode_id=sc.id
WHERE sc.id IS NULL;
I had the same issue with my MySQL database but finally, I got a solution which worked for me.
Since in my table everything was fine from the mysql point of view(both tables should use InnoDB engine and the datatype of each column should be of the same type which takes part in foreign key constraint).
The only thing that I did was to disable the foreign key check and later on enabled it after performing the foreign key operation.
Steps that I took:
SET foreign_key_checks = 0;
alter table tblUsedDestination add constraint f_operatorId foreign key(iOperatorId) references tblOperators (iOperatorId); Query
OK, 8 rows affected (0.23 sec) Records: 8 Duplicates: 0 Warnings: 0
SET foreign_key_checks = 1;
Use NOT IN to find where constraints are constraining:
SELECT column FROM table WHERE column NOT IN
(SELECT intended_foreign_key FROM another_table)
so, more specifically:
SELECT sourcecode_id FROM sourcecodes_tags WHERE sourcecode_id NOT IN
(SELECT id FROM sourcecodes)
EDIT: IN and NOT IN operators are known to be much faster than the JOIN operators, as well as much easier to construct, and repeat.
Truncate the tables and then try adding the FK Constraint.
I know this solution is a bit awkward but it does work 100%. But I agree that this is not an ideal solution to deal with problem, but I hope it helps.
For me, this problem was a little different and super easy to check and solve.
You must ensure BOTH of your tables are InnoDB. If one of the tables, namely the reference table is a MyISAM, the constraint will fail.
SHOW TABLE STATUS WHERE Name = 't1';
ALTER TABLE t1 ENGINE=InnoDB;
This also happens when setting a foreign key to parent.id to child.column if the child.column has a value of 0 already and no parent.id value is 0
You would need to ensure that each child.column is NULL or has value that exists in parent.id
And now that I read the statement nos wrote, that's what he is validating.
I had the same problem today. I tested for four things, some of them already mentioned here:
Are there any values in your child column that don't exist in the parent column (besides NULL, if the child column is nullable)
Do child and parent columns have the same datatype?
Is there an index on the parent column you are referencing? MySQL seems to require this for performance reasons (http://dev.mysql.com/doc/refman/5.5/en/create-table-foreign-keys.html)
And this one solved it for me: Do both tables have identical collation?
I had one table in UTF-8 and the other in iso-something. That didn't work. After changing the iso-table to UTF-8 collation the constraints could be added without problems. In my case, phpMyAdmin didn't even show the child table in iso-encoding in the dropdown for creating the foreign key constraint.
It seems there is some invalid value for the column line 0 that is not a valid foreign key so MySQL cannot set a foreign key constraint for it.
You can follow these steps:
Drop the column which you have tried to set FK constraint for.
Add it again and set its default value as NULL.
Try to set a foreign key constraint for it again.
I'd the same problem, I checked rows of my tables and found there was some incompatibility with the value of fields that I wanted to define a foreign key. I corrected those value, tried again and the problem was solved.
I end up delete all the data in my table, and run alter again. It works. Not the brilliant one, but it save a lot time, especially your application is still in development stage without any customer data.
try this
SET foreign_key_checks = 0;
ALTER TABLE sourcecodes_tags ADD FOREIGN KEY (sourcecode_id) REFERENCES sourcecodes (id) ON DELETE CASCADE ON UPDATE CASCADE
SET foreign_key_checks = 1;
I had this exact same problem about three different times. In each instance it was because one (or more) of my records did not conform to the new foreign key. You may want to update your existing records to follow the syntax constraints of the foreign key before trying to add the key itself. The following example should generally isolate the problem records:
SELECT * FROM (tablename)
WHERE (candidate key) <> (proposed foreign key value)
AND (candidate key) <> (next proposed foreign key value)
repeat AND (candidate key) <> (next proposed foreign key value) within your query for each value in the foreign key.
If you have a ton of records this can be difficult, but if your table is reasonably small it shouldn't take too long. I'm not super amazing in SQL syntax, but this has always isolated the issue for me.
Empty both your tables' data and run the command. It will work.
I was getting this error when using Laravel and eloquent, trying to make a foreign key link would cause a 1452. The problem was lack of data in the linked table.
Please see here for an example: http://mstd.eu/index.php/2016/12/02/laravel-eloquent-integrity-constraint-violation-1452-foreign-key-constraint/
You just need to answer one question:
Is your table already storing data? (Especially the table included foreign key.)
If the answer is yes, then the only thing you need to do is to delete all the records, then you are free to add any foreign key to your table.
Delete instruction: From child(which include foreign key table) to parent table.
The reason you cannot add in foreign key after data entries is due to the table inconsistency, how are you going to deal with a new foreign key on the former data-filled the table?
If the answer is no, then follow other instructions.
I was readying this solutions and this example may help.
My database have two tables (email and credit_card) with primary keys for their IDs. Another table (client) refers to this tables IDs as foreign keys. I have a reason to have the email apart from the client data.
First I insert the row data for the referenced tables (email, credit_card) then you get the ID for each, those IDs are needed in the third table (client).
If you don't insert first the rows in the referenced tables, MySQL wont be able to make the correspondences when you insert a new row in the third table that reference the foreign keys.
If you first insert the referenced rows for the referenced tables, then the row that refers to foreign keys, no error occurs.
Hope this helps.
Make sure the value is in the other table otherwise you will get this error, in the assigned corresponding column.
So if it is assigned column is assigned to a row id of another table , make sure there is a row that is in the table otherwise this error will appear.
you can try this exapmple
START TRANSACTION;
SET foreign_key_checks = 0;
ALTER TABLE `job_definers` ADD CONSTRAINT `job_cities_foreign` FOREIGN KEY
(`job_cities`) REFERENCES `drop_down_lists`(`id`) ON DELETE CASCADE ON UPDATE CASCADE;
SET foreign_key_checks = 1;
COMMIT;
Note : if you are using phpmyadmin just uncheck Enable foreign key checks
as example
hope this soloution fix your problem :)
UPDATE sourcecodes_tags
SET sourcecode_id = NULL
WHERE sourcecode_id NOT IN (
SELECT id FROM sourcecodes);
should help to get rid of those IDs. Or if null is not allowed in sourcecode_id, then remove those rows or add those missing values to the sourcecodes table.
I had the same problem and found solution, placing NULL instead of NOT NULL on foreign key column. Here is a query:
ALTER TABLE `db`.`table1`
ADD COLUMN `col_table2_fk` INT UNSIGNED NULL,
ADD INDEX `col_table2_fk_idx` (`col_table2_fk` ASC),
ADD CONSTRAINT `col_table2_fk1`
FOREIGN KEY (`col_table2_fk`)
REFERENCES `db`.`table2` (`table2_id`)
ON DELETE NO ACTION
ON UPDATE NO ACTION;
MySQL has executed this query!
In my case, I created a new table with the same structure, created the relationships with the other tables, then extracted the data in CSV from the old table that has the problem, then imported the CSV to the new table and disabled foreign key checking and disabled import interruption, all my data are inserted to the new table that has no problem successfully, then deleted the old table.
It worked for me.
I need make a scan with limit and a condition on DynamoDB.
The docs says:
In a response, DynamoDB returns all the matching results within the scope of the Limit value. For example, if you issue a Query or a Scan request with a Limit value of 6 and without a filter expression, DynamoDB returns the first six items in the table that match the specified key conditions in the request (or just the first six items in the case of a Scan with no filter). If you also supply a FilterExpression value, DynamoDB will return the items in the first six that also match the filter requirements (the number of results returned will be less than or equal to 6).
The code (NODEJS):
var params = {
ExpressionAttributeNames: {"#user": "User"},
ExpressionAttributeValues: {":user": parseInt(user.id)},
FilterExpression: "#user = :user and attribute_not_exists(Removed)",
Limit: 2,
TableName: "XXXX"
};
DynamoDB.scan(params, function(err, data) {
if (err) {
dataToSend.message = "Unable to query. Error: " + err.message;
} else if (data.Items.length == 0) {
dataToSend.message = "No results were found.";
} else {
dataToSend.data = data.Items;
console.log(dataToSend);
}
});
Table XXXX definitions:
Primary partition key: User (Number)
Primary sort key: Identifier (String)
INDEX:
Index Name: RemovedIndex
Type: GSI
Partition key: Removed (Number)
Sort key: -
Attributes: ALL
In code above, if I remove the Limit parameter, DynamoDB will return the items that match the filter requirements. So, the conditions are ok. But when I scan with Limit parameter, the result is empty.
The XXXX table, has 5 items. Only the 2 firsts have the Removed attribute. When I scan without Limit parameter, DynamoDB returns the 3 items without Removed attribute.
What i'm doing wrong?
From the docs that you quoted:
If you also supply a FilterExpression value, DynamoDB will return the
items in the first six that also match the filter requirements
By combining Limit and FilterExpression you have told DynamoDB to only look at the first two items in the table, and evaluate the FilterExpression against those items. Limit in DynamoDB can be confusing because it works differently from limit in a SQL expression in a RDBMS.
Also ran into this issue, i guess you will just have to scan the whole table to a max of 1 MB
Scan
The result set from a Scan is limited to 1 MB per call. You can use the LastEvaluatedKey from the scan response to retrieve more results.
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html
You might be able to get what you need by using a secondary index. Using the classic RDB example, customer - order example: you have one table for customers and one for orders. The Orders table has a Key consisting of Customer - HASH, Order - RANGE. So if you wanted to get the latest 10 orders, there would be no way to do it without a scan
But if you create a Global Secondary Index on orders of "Some Constant" -- HASH, Date RANGE, and queried against that index, they query would do what you want and only charge you for the RCUs involved with the records returned. No expensive scan needed. Note, writes will be more expensive, but in most cases, there are many more reads than writes.
Now you have your original problem if you want to get the 10 biggest orders for a day larger than $1000. The query would return the last 10 orders, and then filter out those less than $1000.
In this case, you could create a computed key of Date-OrderAmount, and queries against that index would return what you want.
It's not as simple as SQL, but you need to think about access patterns in SQL too. If if you have a lot of data, you need to create Indexes in SQL or the DB will happily to table scans on your behalf, which will impair performance and raise your costs.
Note that everything I proposed is normalized in the sense that there is only one source of truth. You are not duplicating data -- you are merely recasting views of it to get what you need from DynamoDB.
Bear in mind that the CONSTANT as a HASH s subject to the 10GB per partition limit, so you would need to design around it if you had a lot of active data. For example, depending on your expected access pattern, you could use Customer and not a constant as a HASH. Or use STreams to organize the data (or subsets) in other ways.
Small hack - Iterate till you get the results
lastEvaluatedKey = null;
do {
if(lastEvaluatedKey != null) {
// query or scan data with last evaluated key
} else {
// query or scan data WITHOUT last evaluated key
}
lastEvaluatedKey == key of last item retrieved
} while(lastEvaluatedKey != null && retrievedResultSize == 0); // == 0 or < yourLimit
If the number of items retrieved is 0 and lastEvaluatedKey is not null that means it has scanned or queried the number of rows which match to your limit. (and result size is zero because they didn't match the filter expression)
So right now I'm trying to draw relations between 3 different tables using sqlite I'm relatively new to sqlite but saw that you can draw relations by using foreign keys thus optimizing performance. So right now here is my sql statements creating my tables:
'CREATE TABLE IF NOT EXISTS shifts (
shifts_id primary integer,
shift_base_id integer,
shift_site_id integer)';
'CREATE TABLE IF NOT EXISTS sites (
site_id primary integer,
site_info text,
FOREIGN KEY(site_id) REFERENCES shifts(shift_site_id))';
'CREATE TABLE IF NOT EXISTS bases (
base_id primary integer,
base_info text,
FOREIGN KEY(base_id) REFERENCES shifts(shift_base_id))';
So what I'm trying to do is draw relations between the child tables (sites and bases) with the parent table (shifts) by the id. The problem I'm running into is I'm getting a "foreign key mismatch" error. I read somewhere that in sqlite you can only REFERENCE a primary or unique key within the parent table. The problem with this is that multiple shifts can share the same sites and bases. For instance base_id could equal 1234 and multiple shifts would have shifts_base_id = 1234.
Also there will be times where a base_id in the bases table wont have a matching shift_base_id in the shifts table.
So my question is how to reference this one to many relation between multiple tables? And how to make that relation optional.
You have your relationships backwards. You list the foreign keys in the table that reference the other tables.
CREATE TABLE IF NOT EXISTS shifts (
shifts_id primary integer,
shift_base_id integer,
shift_site_id integer,
FOREIGN KEY (shift_base_id) REFERENCES bases (base_id),
FOREIGN KEY (shift_site_id) REFERENCES site (site_id));