Develop Reference

How can I scrape/download many files from a dynamic webpage with a hidden xhtml section?

I am trying to download every CSV file on this site: http://data.cnmc.es/datagraph/jsp/inf_trim.jsp
Each CSV is in a radio button inside a sub-page (not a different site) of that main page (I believe the term used to describe this type of site is 'dynamic'). Example of location of one of the CSV files:
image
Problem:
When I run the following code in python in jupyter notebook in order to find patterns in the html - so that I can automate the file downloading process - only a small part of the html is retrieved.
Code:
import requests
from bs4 import BeautifulSoup
webpage = 'http://data.cnmc.es/datagraph/jsp/inf_trim.jsp'
response = requests.get(webpage)
html = response.text
print(html)
What it retrieves:
...some code above which I have purposefully removed for post...
<noframes>
<h1>Informe Trimestral</h1>
<p>Aplicación de consulta de los informes que elabora trimestralmente la Comisión Nacional de los Mercados y la Competencia (CNMC)) en relación al mercado de las telecomunicaciones en España</p>
<h2>Trimestral IV 2014</h2>
<p>1. DATOS GENERALES DEL SECTOR</p>
<p>2. INFRAESTRUCTURAS</p>
<p>3. COMUNICACIONES FIJAS</p>
<p>4. COMUNICACIONES MÓVILES</p>
<p>5. SERVICIOS DE TELEVISIÓN Y RADIO</p>
<p>6. SERVICIOS MAYORISTAS</p>
<h3>EL SECTOR</h3>
<h2>COMUNICACIONES FIJAS</h2>
<h2>COMUNICACIONES MÓVILES</h2>
<h2>AUDIOVISUAL</h2>
<p>2014 © CNMC Comisión Nacional de los Mercados y de la Competencia.</p>
</noframes>
...some code below which I have purposefully removed for post...
What I can see using 'right-click > inspect' on the site:
image
As you can see, the 'right-click > inspect' html shown is much more extended (i.e. it includes <a> tags to dynamically navigate the page which the code-retrieved html does not).
How can I fetch ALL of the html, including those <a> tags?
I believe the answer/problem may lie in this line of code (complete guess, have no clue really):
image
FYI: please correct me if my strategy is not good. My intention is to get the full webpage html to find a pattern in the html tags that will allow me to automate the process of clicking on each CSV radio button using the selenium library
Thanks for your time
Jaime

Find the link in the text and convert it to the embed code

I'm doing a study with Angular for news, announcement website.
I'm printing the content as <div [innerHTML]='res.content'></div>. But the contents have youtube links and they appear as text.
I am;
In the content of the 'div' element, I want to determine the links in youtube and convert them to 'embed code'.
An example res.content from Api;
"<p style=\"text-align: center;\"><strong>test yazıdır.</strong></p>\r\n \r\n\r\n<b> </b>\r\n<h4>project description</h4>\r\nMaturity is required as young people will look to the volunteer (&\r\nstaff in general) as a role model.\r\n<div class=\"clearfix\"></div>\r\n<h4>how to apply</h4>\r\nSend CV and covering letter to <strong>info#galwaysteinerschool.com</strong> and name of sending organisation where applicable\r\n\r\n \r\n\r\n<strong>Son Başvuru Tarihi:</strong> 01/02/2019\r\n\r\n \r\n\r\n \r\n\r\n \r\n\r\nhttps://www.youtube.com/watch?v=sgUVUVY2e-I\r\n\r\n \r\n\r\n \r\n\r\n \r\n\r\n<strong>EVS gönderici kuruluş listesi ve başvuru ile ilgili makalemize ulaşmak için tıklayınız.</strong>\r\n\r\n<strong>Diğer AGH/EVS ilanlarına ulaşmak için tıklayınız.</strong>\r\n\r\n<strong>Motivasyon Mektubu Nasıl Yazılır makalemizi okumak için <a title=\"Motivasyon Mektubu Nasıl Yazılır?\" href=\"http://www.example.com/motivasyon-mektubu-nasil-yazilir/\" target=\"_blank\" rel=\"noopener\">tıklayınız.</a></strong>\r\n\r\n<strong>Blog yazarımız Seher Can’ın Avrupa Gönüllü Hizmeti deneyimi makalesi için tıklayınız.</strong>\r\n\r\n<strong>Blog yazarımız Emre Gögebakan’ın Avrupa Gönüllü Hizmeti başvuru süreci makalesi için tıklayınız.</strong>\r\n\r\n<strong>AGH/EVS mülakatı teknikleri makalemiz için tıklayınız.</strong>\r\n\r\n<strong>AGH EN ÇOK SORULAN SORULAR VE CEVAPLARI makalemiz için tıklayınız.</strong>\r\n\r\n<strong>Başarılı Motivasyon Mektubu Teknikleri makalemiz için tıklayınız.</strong>\r\n\r\n \r\n\r\n \r\n\r\n<strong>Kaynak:</strong> https://europa.eu/youth/volunteering/project/9729_en\r\n\r\n \r\n<p style=\"text-align: center;\"><strong>Bu ilan Avrupa Birliği resmi sitesinden duyuru amaçlı kaynak gösterilerek Türkçe diline çevrilmiştir. İlan sorumluluğu example.com sitesinde değildir. Özgeçmiş ve motivasyon mektubu hazırlanması işlemlerinde ÜCRETSİZ olarak kurumumuz destek vermektedir. Kurumumuzun başvurunun olumlu veya olumsuz sonuçlanması konusunda herhangi bir taahhüdü ve etkisi yoktur.</strong></p>";
What kind of a method should I apply for this? and how can I?
What I want to do exactly is find the youtube url inside the article and automatically translate it into the youtube embed code.
Thank you from now.

You could use regex to get the video id from the youtube link.
Then all you need is to add it to the youtube embeded link.
So if you use regex to find the video link from the following link:
https://www.youtube.com/watch?v=sgUVUVY2e-I\
it would return you with this: sgUVUVY2e-I
Then you can insert that into the embeded link like this:
https://www.youtube.com/embed/sgUVUVY2e-I
There are a tone of tutorials on using regex to find information inside a text, then it is all up to you how you want to dislay it.

ExpressionEngine textarea not supporting embedded code

I have installed the ExpressionEngine 2.11. and I have created a Channel Field of a textarea type to post the full description of my articles.
My issue comes when trying to include Twitter or Instagram embed code like this one
<blockquote class="twitter-tweet" data-cards="hidden" data-lang="es"><p lang="es" dir="ltr">🏈 SÚPER QUINIELA #NFL 🏈<br><br>¡Gana tarjetas de regalo y viajes a partidos oficiales!<br><br>Juega 👉 t.co/SVMJMjRaHZ pic.twitter.com/MIkMkRdCAa</p>— Innovasport (#innovasport) 1 de septiembre de 2016</blockquote>
<script async src="platform.twitter.com/widgets.js" charset="utf-8"></script>
I've got the following error
Any idea how to let textarea fields to support embeddeds HTML characters?
Greetings, Leo

Joomla Modal box popup code works in article, but not in code

This question goes for Joomla developers, or enthusiasts.
Unfortunately there is no way to reproduce this scenario on js fiddle website.
I have the following code:
<!-- AQUI COMEÇA O MODAL POPUP, QUE É O POPUP BUILTIN DO JOOMLA AO ESTILO LIGHTBOX -->
<!-- Esta parte ativa a função -->
<?php JHTML::_('behavior.modal', 'a.modal'); ?><!-- Esta parte utiliza javascript para transformar o layout do popup em um fluido -->
<script>jQuery(document).ready(function($){
var width = $(window).width();
var height = $(window).height();
//ID of container
$('a#modal_info').attr('rel','{handler: "iframe", size: {x: '+(width-(width*0.03))+', y: '+(height-(height*0.20))+'}}');
});</script>
<!-- Esta parte é o Link para fazer a chamada -->
<div class="modal barradofundo" onclick="window.location.href = this.getElementsByTagName('a')[0].href;">
<div id="bf1"><img id="bf1img" src="/images/images201605/Coin-icon-e.png" alt="Dinheiro" border="0"></div>
<!-- Esta parte é o Link para fazer a chamada -->
<div id="bf2"><span>Saia da crise agora mesmo, destaque-se e ganhe muito dinheiro!</span>
</div>
<div id="bf3"><a id="modal_info" href="/va/filetobeloaded.html" class="modal catalogobotao2">SAIBA +</a></div>
</div>
<div class="barradofundoespaco"></div>
What that code does is to open a modal box popup that uses a fluid a layout, in other words, it's like a lightbox that fits window.
Inside that box it loads an external page.
The {source} tag is from Sourcerer Plugin, which is used to load javascript, HTML or php inside any part of the site.
It works just fine when used inside an article, but when I try using it in a custom module, it doesn't work, it not even works on other modules that accept php and HTML.
I have jQuery enabled, properly configured and have not conflicts showing in the console. It looks like it's a problem with the module.

Temporary solution:
Found out that using the <?php JHTML::_('behavior.modal', 'a.modal'); ?> in the index.php file of the template, I was able to put it to work. But the problem is that I can't hack the code, as I'll lose the hack after a system's update.

AngularJS does not translate correctly with translate-value

I got a small problem with my I18n integration.
AngularJS seems to skip one translation in the whole page. Everything translates fine except for one tag.
I get my translations through JSON.
{
"password": {
"title": "Wachtwoord voor [<b>{{username}}</b>]",
"form": {
"button": "Opslaan"
},
"messages": {
"error": "<strong>Er is een fout opgetreden!</strong> Het wachtwoord kon niet worden veranderd.",
"success": "<strong>Wachtwoord veranderd!</strong>"
}
}
}
The line in the HTML:
<h2 translate="password.title" translate-values="{username: '{{account.login}}'}">Password for [<b>{{account.login}}</b>]</h2>
This just displays Password for [Admin].
The request for the password.json file contains exact the same as i stated above. The html contains this when inspected through the Chrome Console:
<h2 translate="password.title" translate-values="{username: 'admin'}" class="ng-scope">Password for [admin]</h2>
Everything else (Without those translate-values) translate fine. But this line doesnt. Why doesnt it work? The login name is correctly passed.
Thanks in advance.
Cheers!

If you use the translate directive, there is no need to add content inside the h2 tag. So, the following should work :
<h2 translate="password.title" translate-values="{username: '{{account.login}}'}"></h2>