MathJAX Typeset Queue Error when loading new data into DOM (Cannot read property 'contains' of null)

My website uses a javascript function to load (via AJAX) math markup into the DOM and then render it with MathJAX.
If the function is called a second time (i.e., to load some new math markup into a DOM element):

All works well if MathJAX has finished processing its Typeset queue prior to the second function call.

An error appears if MathJAX has NOT finished processing its Typeset queue prior to the second function call.
MathJax.js?config=TeX-MML-AM_CHTML&latest:19 Uncaught TypeError: Cannot read property 'contains' of null

Once the error occurs, MathJAX is in a bad state and will not function until entirely reloaded (e.g., via a page reload).
It seems like the easiest solution to this problem would be to clear the MathJAX Typeset queue before loading new math via AJAX, but I can find no method in the MathJAX API that clears the Typeset queue.
Here's a stripped-down version of the function (using jQuery syntax) I'm using:
window.loadNewMath = function() {
	// I want to clear the MathJAX Typeset queue before making the AJAX call

	$.ajax({
		dataType: "json",
		type: "POST",
		url: "/getNewMath",
		data: [],
		success: function(data) {
			if (data['success']) {
				// load math into DOM
				$("#math-panel").html(data.new_math);
				MathJax.Hub.Queue(["Typeset", MathJax.Hub, 'math-panel']);
			} else {
				$("#math-panel").html('oops. there was a database error');
			}
		},
		error: function(data) {
			$("#math-panel").html('oops. there was an ajax error');
		}
	})
}

This scenario seems like a common use case, so I hope someone there is a way of avoiding this error.

You are right that there is an issue if you make the second function call before MathJax has finished typesetting the first one.  That is because you are replacing the portion of the DOM where MathJax is typesetting the math, so you have pulled rug out from under MathJax by removing the DOM nodes that it is working on (MathJax does retain pointers to some DOM nodes that is working with, so those will not be freed by the browser, but the parent nodes will, and that is a problem.
You are asking about clearing the "typeset" queue, but the queue is about more than just typesetting, so clearing the queue is not the right action (and even if you did, that doesn't stop the current action in the queue).  You probably mean to ask about canceling a typesetting action that is already initiated, but you are right, there is no API for that (though it is possible to configure one).  But even if there were, you would still be in trouble, because such a cancelation would not be immediate (things are operating asynchronously), so you would still ned to be careful how that was done.
The real issue is that you are not synchronizing the changing of the DOM with the typesetting operations.  Once you have started a typesetting operation, you should not modify the DOM (at least where the typesetting is occurring) until the typesetting is done.  One way to accomplish that is to queue your changes to the DOM as well as the typesetting.  That way, the changes will be put off until any previous typesetting is finished.  For example,
$.ajax({
	dataType: "json",
	type: "POST",
	url: "/getNewMath",
	data: [],
	success: function(data) {
		if (data['success']) {
			// load math into DOM
			MathJax.Hub.Queue(
				function () {$("#math-panel").html(data.new_math);},
				["Typeset", MathJax.Hub, 'math-panel']
			);
		} else {
			$("#math-panel").html('oops. there was a database error');
		}
	},
	error: function(data) {
		$("#math-panel").html('oops. there was an ajax error');
	}
})

should resolve the timing issue.
Note, however, that your ajax call may complete before the change to the DOM (and subsequent typeset) occurs.  So if your ajax call could be made more quickly than MathJax can typeset the changes, this could lead to longer and longer delays between the requested changes and the resulting output on screen.  In that case, you might need to be more sophisticated about your handling of the typeset calls.  One way would be to keep track of whether a typeset operation is in progress and  not queue additional ones until after the typeset is complete; keep track of the last new_math that has been requested (no need to typeset other ones that might have been requested while an earlier one was typeset), and when the typesetting has finished, set the new data and typeset again.  You should be able to work out the logic for that.
The best solution, however, might be to disable whatever GUI controls allow the user to request new math during the time that MathJax is processing.  Something like
MathJax.Hub.Queue(
	DisableGui,
	["Typeset", MathJax.Hub, 'math_panel'],
	EnableGui
);

where DisableGui() and EndableGui() are functions to disable/enable the controls should do the trick.
Davide

On Oct 24, 2018, at 7:03 PM, XYZ MathTV Web Development<EMAIL_ADDRESS>wrote:
My website uses a javascript function to load (via AJAX) math markup into the DOM and then render it with MathJAX.
If the function is called a second time (i.e., to load some new math markup into a DOM element):
All works well if MathJAX has finished processing its Typeset queue prior to the second function call.
An error appears if MathJAX has NOT finished processing its Typeset queue prior to the second function call.
MathJax.js?config=TeX-MML-AM_CHTML&latest:19 Uncaught TypeError: Cannot read property 'contains' of null
Once the error occurs, MathJAX is in a bad state and will not function until entirely reloaded (e.g., via a page reload).
It seems like the easiest solution to this problem would be to clear the MathJAX Typeset queue before loading new math via AJAX, but I can find no method in the MathJAX API that clears the Typeset queue.
Here's a stripped-down version of the function (using jQuery syntax) I'm using:
window.loadNewMath = function() {
// I want to clear the MathJAX Typeset queue before making the AJAX call
$.ajax({
dataType: "json",
type: "POST",
url: "/getNewMath",
data: [],
success: function(data) {
if (data['success']) {
// load math into DOM
$("#math-panel").html(data.new_math);
MathJax.Hub.Queue(["Typeset", MathJax.Hub, 'math-panel']);
} else {
$("#math-panel").html('oops. there was a database error');
}
},
error: function(data) {
$("#math-panel").html('oops. there was an ajax error');
}
})
}
This scenario seems like a common use case, so I hope someone there is a way of avoiding this error.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/mathjax/MathJax/issues/2071, or mute the thread https://github.com/notifications/unsubscribe-auth/AAaajvAc6Hf8FRYiWtslzcIsC0zA0-H7ks5uoPG6gaJpZM4X5NWw.

You are right that there is an issue if you make the second function call before MathJax has finished typesetting the first one.  That is because you are replacing the portion of the DOM where MathJax is typesetting the math, so you have pulled rug out from under MathJax by removing the DOM nodes that it is working on (MathJax does retain pointers to some DOM nodes that is working with, so those will not be freed by the browser, but the parent nodes will, and that is a problem.
You are asking about clearing the "typeset" queue, but the queue is about more than just typesetting, so clearing the queue is not the right action (and even if you did, that doesn't stop the current action in the queue).  You probably mean to ask about canceling a typesetting action that is already initiated, but you are right, there is no API for that (though it is possible to configure one).  But even if there were, you would still be in trouble, because such a cancelation would not be immediate (things are operating asynchronously), so you would still ned to be careful how that was done.
The real issue is that you are not synchronizing the changing of the DOM with the typesetting operations.  Once you have started a typesetting operation, you should not modify the DOM (at least where the typesetting is occurring) until the typesetting is done.  One way to accomplish that is to queue your changes to the DOM as well as the typesetting.  That way, the changes will be put off until any previous typesetting is finished.  For example,
$.ajax({
	dataType: "json",
	type: "POST",
	url: "/getNewMath",
	data: [],
	success: function(data) {
		if (data['success']) {
			// load math into DOM
			MathJax.Hub.Queue(
				function () {$("#math-panel").html(data.new_math);},
				["Typeset", MathJax.Hub, 'math-panel']
			);
		} else {
			$("#math-panel").html('oops. there was a database error');
		}
	},
	error: function(data) {
		$("#math-panel").html('oops. there was an ajax error');
	}
})

should resolve the timing issue.
Note, however, that your ajax call may complete before the change to the DOM (and subsequent typeset) occurs.  So if your ajax call could be made more quickly than MathJax can typeset the changes, this could lead to longer and longer delays between the requested changes and the resulting output on screen.  In that case, you might need to be more sophisticated about your handling of the typeset calls.  One way would be to keep track of whether a typeset operation is in progress and  not queue additional ones until after the typeset is complete; keep track of the last new_math that has been requested (no need to typeset other ones that might have been requested while an earlier one was typeset), and when the typesetting has finished, set the new data and typeset again.  You should be able to work out the logic for that.
The best solution, however, might be to disable whatever GUI controls allow the user to request new math during the time that MathJax is processing.  Something like
MathJax.Hub.Queue(
	DisableGui,
	["Typeset", MathJax.Hub, 'math_panel'],
	EnableGui
);

where DisableGui() and EndableGui() are functions to disable/enable the controls should do the trick.

Thank you for your detailed reply!
I think I will indeed need to disable GUI controls until the typesetting is complete (at least until I can find a better solution).
Just for the record, I think this is a pretty common use case:

user clicks UI button that loads content including math markup via AJAX call and writes it to a DOM element; MathJAX typesetting is initiated;
user clicks another UI button to load different content including math markup via AJAX call and writes it to the same DOM element; MathJAX typesetting is initiated;

In our case, the user is navigating the sections of an online math textbook (e.g.: https://www.xyztextbooks.com/ebook/title/college_algebra), and forcing them to wait for all Math to be typeset by MathJAX before they can navigate to another section is not a good solution. We have (in another project) used KaTeX which allows us to pre-render the Math on a server (running NodeJS), and I suppose we could move in that direction (though I'd rather stick with MathJAX).
I'm interested in what you mean when you say "there is no API for that (though it is possible to configure one)". I would dearly like to have a method for clearing out pending MathJAX tasks prior to modifying the DOM.

We have (in another project) used KaTeX which allows us to pre-render the Math on a server (running NodeJS), and I suppose we could move in that direction (though I'd rather stick with MathJAX).

MathJax v3 (currently in beta) is a complete rewrite of MathJax, and it currently operates synchronously (so no queuing needed).  You may find that this is easier to work with in the long rung (though there are still features that are being worked on).  It also operates in NodeJS as easily as in a browser, so that could be used for back-end processing.
There is also the mathjax-node project that is a server-side implementation of MathJax v2.  That is something you could use now if pre-processing by MathJax is a goal for you.

I would dearly like to have a method for clearing out pending MathJAX tasks prior to modifying the DOM.

What I had in mind was a means of telling MathJax to cancel the current typeset operation (which is not the same as clearing the queue, but is more likely to be what you actually need).  I wrote the following extension for another project, which implements that:
(function () {
    var HUB = MathJax.Hub;

    if (!HUB.Cancel) {

        HUB.cancelTypeset = false;
        var CANCELMESSAGE = "MathJax Canceled";

        HUB.Register.StartupHook("HTML-CSS Jax Config", function () {
            var HTMLCSS = MathJax.OutputJax["HTML-CSS"],
                TRANSLATE = HTMLCSS.Translate.bind(HTMLCSS);
            HTMLCSS.Augment({
                Translate: function (script, state) {
                    if (HUB.cancelTypeset || state.cancelled) {
                        throw Error(CANCELMESSAGE)
                    }
                    return TRANSLATE(script, state);
                }
            });
        });

        HUB.Register.StartupHook("SVG Jax Config", function () {
            var SVG = MathJax.OutputJax["SVG"],
                TRANSLATE = SVG.Translate.bind(SVG);
            SVG.Augment({
                Translate: function (script, state) {
                    if (HUB.cancelTypeset || state.cancelled) {
                        throw Error(CANCELMESSAGE)
                    }
                    return TRANSLATE(script, state);
                }
            });
        });

        HUB.Register.StartupHook("CommonHTML Jax Config", function () {
            var CHTML = MathJax.OutputJax.CommonHTML,
                TRANSLATE = CHTML.Translate.bind(CHTML);
            CHTML.Augment({
                Translate: function (script, state) {
                    if (HUB.cancelTypeset || state.cancelled) {
                        throw Error(CANCELMESSAGE);
                    }
                    return TRANSLATE(script, state);
                }
            });
        });

        HUB.Register.StartupHook("PreviewHTML Jax Config", function () {
            var PHTML = MathJax.OutputJax.PreviewHTML,
                TRANSLATE = PHTML.Translate.bind(PHTML);
            PHTML.Augment({
                Translate: function (script, state) {
                    if (HUB.cancelTypeset || state.cancelled) {
                        throw Error(CANCELMESSAGE);
                    }
                    return TRANSLATE(script, state);
                }
            });
        });

        HUB.Register.StartupHook("TeX Jax Config", function () {
            var TEX = MathJax.InputJax.TeX,
                TRANSLATE = TEX.Translate.bind(TEX);
            TEX.Augment({
                Translate: function (script, state) {
                    if (HUB.cancelTypeset || state.cancelled) {
                        throw Error(CANCELMESSAGE)
                    }
                    return TRANSLATE(script, state);
                }
            });
        });

        var PROCESSERROR = HUB.processError.bind(HUB);
        HUB.processError = function (error, state, type) {
            if (error.message !== CANCELMESSAGE) {
                return PROCESSERROR(error, state, type)
            }
            MathJax.Message.Clear(0, 0);
            state.jaxIDs = [];
            state.jax = {};
            state.scripts = [];
            state.i = state.j = 0;
            state.cancelled = true;
            return null;
        };

        HUB.Cancel = function () {
            this.cancelTypeset = true;
        };
    }
})();

This implements a function MathJax.Hub.Cancel() that tells MathJax to stop typesetting at the earliest opportunity.  Note that it is not necessarily immediate, but as soon as possible.  So you still need to use the queue to manage your DOM changes, but you should not have to wait for the entire previous DOM to be processed.
You can also keep track of whether MathJax currently is typesetting, and hold off on changes until the cancel goes through.  Something like the following (untested) code:
var mathPending = false;
var newMath = '';

function TypesetPendingMath() {
  if (mathPending) {
    $("#math-panel").html(newMath);
    newMath = '';
    mathPending = false;
    MathJax.Hub.cancelTypeset = false;
    MathJax.Hub.Queue(
        ["Typeset", MathJax.Hub. 'math-panel'],
        TypesetPendingMath
     );
  }
}

function ChangeMath(dom) {
  MathJax.Hub.Cancel();
  newMath = dom;
  if (!pendingMath) {
    pendingMath = true;
    MathJax.Hub.Queue(TypesetPendingMath);
  }
}

This provides a function ChangeMath() that allows you to specify new HTML containing math content.  If there is no typesetting currently going on, the DOM will be modified and typeset.  Otherwise, if we are in the middle of typesetting, we cache the change and cancel the current typesetting (as soon as possible).  If the math changes again before previous typesetting is done, the old change is discarded and the new one cached.
The key to making it work is the TypesetPendingMath() function, which checks if there was cached math to be typeset, and if so, it sets the DOM to the cached math, clears the pending and cancel flags, then starts typesetting the math.  The critical step is that it also queues another call to TypesetPendingMath().  If the user hasn't done anything while the typesetting is occurring, when the typesetting is finished, the pendingMath flag is false, and nothing more needs to be done.  But if the user requested a change in the math, the pendingMath math flag will have been set (and the math cached in newMath) and the current typesetting operation would have been requested to cancel.  In that case, the typesetting will stop as soon as it can, and the queued TypesetPendingMath() call will run.  Since pendingMath is true in this case, the cached math will be put in place, the flags will be cleared, and it will be typeset.  Afterward, we check if another change has been requested, and make the change and typeset again, otherwise, we just end.
Although I haven't testing this specific incarnation of the code, I have certainly used this technique in the past, and it should get you on the right path, even if it is not quite right as it stands.

PS, if you don't want to support all the output formats, you can remove the unwanted blocks from the cancel code above.

Thanks, again, for taking time to provide a detailed analysis of the situation!
I'm currently testing a much simpler solution that seems to be working:

make ajax call to get new math markup
disable navigation controls in the UI
write the new Math markup to the DOM
add the Typeset call to the MathJAX Queue and leverage the Typeset's callback to re-enable the UI controls when typesetting is complete.

Every time I've used this code, and I monitor MathJAX signals in the browser's javascript console, the events occur in the order I want (i.e., typesetting tasks are all completed and then the callback to re-enable UI controls is run.
Here's sample code:
`
window.loadNewMath = function() {
// I want to clear the MathJAX Typeset queue before making the AJAX call
$.ajax({
	dataType: "json",
	type: "POST",
	url: "/getNewMath",
	data: [],
	success: function(data) {
		if (data['success']) {

			// disable navigation controls in UI (they are reenabled after mathjax processing, if any)
			enableEbookNavigationControls(false);

			// load math into DOM
			$("#math-panel").html(data.new_math);

			// queue MathJAX Typeset (with callback to re-enable the UI controls)
			MathJax.Hub.Queue(
				function() {
					MathJax.Hub.Typeset('video-panel', function() {
						// re-enable nav controls when typesetting is complete
						enableEbookNavigationControls(true);
					});
				}
			);
		} else {
			$("#math-panel").html('oops. there was a database error');
		}
	},
	error: function(data) {
		$("#math-panel").html('oops. there was an ajax error');
	}
})

}
`

Yes, your solution should properly synchronize the changes with MathJax.  It is essentially what I had recommended at the end of my first response above; I had suggested pushing the re-enable command on the queue rather than use the Typeset callback, but these are essentially equivalent. So you could do this without the additional closure via
      if (data['success']) {
        enableEbookNavigationControls(false);
        $("#math-panel").html(data.new_math);
        MathJax.Hub.Queue(
          ["Typeset", MathJax.Hub, 'video-panel'],
          [enableEBookNavigationControls, true]
        );
      }

Hmm.. Right you are! In my early tests, I could have sworn that the syntax you recommended resulted in things happening in an undesired order, but I just re-ran the tests (observing the signals), and it all works as predicted.
Thanks again for all your help!

No problem.  Thanks for reporting your findings.  I'm closing this issue, then.


update / 6

done 6.

Hey @marcos-jesus, could you please take a look here - https://github.com/ParanoidUser/yolo/discussions/5


[GOBBLIN-1388] Fix bug when comparing partition type

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-1388

Description

[x] Here are some details about my PR, including screenshots (if applicable):
In this FSDatasetPartitionConfig method, there is a bug where this.partitionType is just being compared to itself (always returning true), instead of compared to other.partitionType.

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:
Trivial

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Codecov Report

Merging #3227 (8fbb510) into master (5a2fb0d) will decrease coverage by 37.33%.
The diff coverage is 0.00%.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5a2fb0d...8fbb510. Read the comment docs.

LGTM 👍  thanks for quick fix


100% cpu usage when running in background

ttyd without any IO is eating 100% of cpu
running in background: ttyd -u 1000 -c my_user:my_password -p **** /bin/bash >/var/log/my_log 2>&1 &
not reproducible - sometimes happen also when the client is idle (no IO) ...
Log Output:

ttyd 1.6.1-fc037fe (libwebsockets 2.0.3)
tty configuration:
credential: **************
start command: /bin/bash
close signal: SIGHUP (1)
terminal type: xterm-256color
Initial logging level 7
Libwebsockets version: 2.0.3 unknown-build-hash
IPV6 not compiled in
libev support compiled in but disabled
libuv support compiled in and enabled
Threads: 1 each 1024 fds
mem: platform fd map:  4096 bytes
Compiled with OpenSSL support
SSL disabled: no LWS_SERVER_OPTION_DO_SSL_GLOBAL_INIT
Creating Vhost 'default' port ****, 2 protocols
Listening on port ****
mem: per-conn:          600 bytes + protocol rx buf
canonical_hostname = s1
Set privs to user '*****'
Listening on port: *****
Using foreign event loop...
lws_protocol_init
HTTP / - <IP_ADDRESS>
HTTP /token - <IP_ADDRESS>
WS   /ws - <IP_ADDRESS>, clients: 1
started process, pid: 8975
Environment:

OS: Debian stretch 9.13
Browser: [Firefox 81.0 Win64]

Can't reproduce, feel free to reopen if you can reproduce it.


Update hidden.css

Hid titlebar-inner to fix the preview/edit/close buttons icon not lining up with the clickable link. Issue #53

I'm not a real developer and if I submitted this pull request wrong I'm sorry.   Please let me know and I'll fix it.


RED-8 experiments admin panel

Overview
This PR implements read-only part of the experiments dashboard feature. The feature to make changes will be part of another PR.
Dashboard is implemented in the admin, on a regular changelist admin view.
Details
Testing instructions
Freely ping fhrzenjak on Slack to assist with practicals.
settings
Locally it is possible to try out the code on either MatchingTool or on main site, by having it serve as both "client" and "server". Not much sense for that use case, but quite good for review. (In real usage, the main site will indeed function this way, talking to itself via API.)
To enable this, add some local settings:
EXPERIMENTS_API = {
    'api_mode': 'server, client',
    'remotes': [
        {
            'url': 'http://localhost:8000',   # or 8001
            'token': '<A TOKEN FOR API>',
        },
    ],
    'local': {
        'name': 'Main site',  # or 'Matching Tool'
        'page_size': 100,
    }
}

getting the code
Install this version of django-experiments by running:
pip uninstall consumeraffairs-django-experiments
pip install consumeraffairs-django-experiments==1.4.0rc1

Tests

[x] Test 1. - check that the API is up

Log in as an admin user
Open /experiments/api/, and verify that:

"browsable API" interface is rendered
the representation contains a link to v1 of the API

Click on the link to v1, and verify that:

this representation contains a link to list of all experiments on this site

Click on the link, and verify that:

the results really contain all experiments on the site

Click on link under aurl field, and verify that:

the representation shows only the selected experiment
state of the experiment can be changed using the form for PUT requests at the bottom of the page

[ ] Test 2. - check that admin dashboard is up

Open /admin/experiments/remoteexperiment/ and verify that:

all the experiments on the site are listed here
if you are testing with multiple sites configured, the list should contain experiments from all sites:

at this point, all confidence intervals should be either no alternatives or no primary goals

Open an experiment with says no primary goals

Scroll down to Primary Goals section (on the main additionally click expand link to see the table)

Mark Pri checkbox (stands for "primary") next to one of already marked goals.

Click Save and continue editing and verify that:

in the "Results" table (at the top of the page) the selected goal in now marked with bold fontface
take note of the confidence interval values for that goal:

Go back to /admin/experiments/remoteexperiment/, reload, and verify that:

the edited experiment now shows a mini table under confidence intervals with values from previous step:

@frnhr : on Test 1 Step 4I get the follwoing error http://sentry.consumeraffairs.com/consumeraffairs/qa/issues/29538/
Pls advise

LGTM on QA


test only

Overview

Testing recommendations

Related Issues or Pull Requests

Checklist

[ ] Code is written and works correctly
[ ] Changes are covered by tests
[ ] Any new configurable parameters are documented in rel/overlay/etc/default.ini
[ ] Documentation changes were made in the src/docs folder
[ ] Documentation changes were backported (separated PR) to affected branches

noting that Content-MD5 header cannot be generalised to other digest algorithms.

@jiahuili430 what was the reason for closing this PR?  I think I know but I am curious.

@jiahuili430 what was the reason for closing this PR? I think I know but I am curious.

Nick and Bob have a better solution to replace the md5-related code, so I closed this one.

Use xxHash: https://github.com/apache/couchdb/pull/4570
Remove Content-MD5 header support: https://github.com/apache/couchdb/pull/4574


Support for sending customized format data

Currently, only u32 types are supported for sending data:
https://github.com/spacewalkhq/raft-rs/blob/5853c9f2461fdde76ef002e2246f73a3c93a04d5/src/server.rs#L644-L651
The effect I'm hoping to achieve is to be able to send any type;
My idea is that this can be achieved using trait.

great idea.


Filter View does not work on SQL Table Aggregate fields

Describe the bug
If a table is created using the 'SQL query' type and it contains a field generated by an aggregate function (i.e. COUNT, SUM, etc), attempting to use a 'Filter view' template on the field to filter a 'List view' results in an error indicating the column does not exist.
"In TestList view (List viewtemplate): column 'sum' does not exist.
To Reproduce
Steps to reproduce the behavior:

Create a 'Database table' test_table with two fields Field1:string, Field2:integer
Populate the test_table with several rows.
Create a 'SQL query' table test_sql with the following sql SELECT field1, SUM(field2) FROM test_table GROUP BY field1;
Create a 'List View' TestList using test_sql table
Create a 'Filter View' TestFilter using test_sql table with a 'DropDownFilter' for field1 and sum fields.
Create a 'Page' TestPage with Views TestFilter and TestList.
View the TestPage
Select the Drop Down filter for field1.  It works as expected.
Select the Drop Down filter for sum.  It fails with the errorIn TestList view (List viewtemplate): column "sum" does not exist

Expected behavior
The list view should behave the same for 'field1' and 'sum'.
Screenshots

Additional Content
I reproduced with version 1.0.0 and 1.1.0-beta.11

Additional information:
If I eliminate the 'TestFilter' above and just place the TestList on a page and set the 'Extra state Formula' to {sum: 0}, when attempting to view the page I receive a similar error indicating a column named 'sum' does not exist even though it does exist.


[file_crash_bug] Bad Request for url: https://bugzilla-dev.allizom.org/rest/bug

2023-07-05 00:42:28,430 - ERROR - Rule file_crash_bug
Traceback (most recent call last):
  File "/home/rm_bot/bugbot/bugbot/bzcleaner.py", line 844, in run
    self.send_email(date=date)
  File "/home/rm_bot/bugbot/bugbot/bzcleaner.py", line 763, in send_email
    data = self.get_email_data(date)
  File "/home/rm_bot/bugbot/bugbot/bzcleaner.py", line 681, in get_email_data
    bugs = self.get_bugs(date=date)
  File "/home/rm_bot/bugbot/bugbot/rules/file_crash_bug.py", line 192, in get_bugs
    resp.raise_for_status()
  File "/home/rm_bot/bugbot/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://bugzilla-dev.allizom.org/rest/bug

The following is the error message from Bugzilla:
{
    "code": 51,
    "message": "There is no component named 'Audio/Video: Web Codecs' in the 'Core' product.",
    "error": true,
    "documentation": "http://www.bugzilla.org/docs/4.2/en/html/api/"
}

This is because the components on bugzilla-dev.allizom.org are not up-to-date with the ones on bugzilla.mozilla.org.

The Core::Audio/Video: Web Codecs component has been added to bugzilla-dev.allizom.org (thanks to @dklawren)


Minimal Theme's Default CSS Classes do not work

Describe the bug
I was trying to use minimal themes predefined css class list-cards using this plugin but it did not work.. Anyway it works with css-classes I define.. What can be done to make it work...
Screenshots
https://github.com/user-attachments/assets/6388e2e8-4d5b-4922-91c4-adff15d4aae2
Device (please complete the following information):

Operating System: Linux
Obsidian version: v1.7.7

By the way this is the CSS Classes defined by Minimal Theme.. I tried a manual css snippet. It also did not work..
https://github.com/kepano/obsidian-minimal/blob/4ae34e15b9fc37d6505a22a15c5f4f47993eec1c/src/scss/features/cards.scss#L273


rub error

When i switch currency to RUB, wallet can't connect to Emerald Service
usd screenshot
rub screenshot

We currently do not provide conversion to RUB, but it's a bug that it's shown as a connection error.  Thank you for reporting the issue.

Fixed on server, will be available on the wallet (i.e. fixed) in the following days

Rolled to production servers, RUB should be working now


Add note about Mise version manager

See https://github.com/Shopify/ruby-lsp/issues/2170#issuecomment-2162983705

I've updated it to point to the Mise docs instead. I think the locality of the advice is important - people tend to skim docs, so having this next to the Mise topics makes it more discoverable, rather than as a general statement further up.


Refreshing workflow where default version has been deleted on GitHub throws an error

Describe the bug
A user tried refreshing a workflow but got the following error.
The webservice encountered an error.

[HTTP 500] OK: org.hibernate.exception.ConstraintViolationException: could not execute statement could not execute statement ERROR: update or delete on table “workflowversion” violates foreign key constraint “fk_workflow_default_workflowversion” on table “workflow” Detail: Key is still referenced from table “workflow”.

It seems like they had a version set as default, but the version was deleted on GitHub.
See discussion for more information.
https://discuss.dockstore.org/t/error-on-refreshing-workflow-update-or-delete-on-table-workflowversion-violates-foreign-key-constraint/3825
To Reproduce
Steps to reproduce the behavior:

Have a workflow with the default version set
Delete the version on GitHub
Refresh the workflow on Dockstore, see error.

Expected behavior
Should delete the default version and then set another as default as a fallback.
┆Issue is synchronized with this Jira Story
┆Fix Versions: Dockstore 1.11
┆Issue Number: DOCK-1545
┆Sprint: Sprint 59- Hammerhead shark
┆Issue Type: Story

This has already been fixed

Confirmed this is fixed in dev
Webservice - 38d2dd5
UI - 2.7.3-73-gf750256a
Compose Setup - 36c8ddc
Deploy - 1.10-58-g820f90c


ignore output directory target after building

considering that many developer may use maven to rebuild their local project and then produce the output directory 'target', so add it to .gitignore

This won't be applied as a) Tomcat doesn't use Maven to build and b) the comment in the file


Fix tests for UI toolkit

The Chore
Importing ui toolkit components into the app at the moment prevents jest from running. Possibly due to the bundling of UI toolkit in the frontend monorepo.

@dexturr did you say you'd like to tackle all the ui-toolkit issues as part of this? That would include

Ensuring components that use ui-toolkit are testable
Ensure the css is bundle in a way that takes advantage of the benefits of tailwind but also makes installation of the ui-toolkit easy
Fix the unnecessary ui-toolkit deps


Question related to swallowing non fetch related exceptions in StateSources

We had a funky circular reference issue just a while ago and the error would bubble up through the failed status of fetch. Basically the action creator was undefined and the call to UserSourceActionCreators.receiveUser(res.body); obviously failed.
However, this was an issue with our code and not an exception in the flow itself. Should those kind of exceptions be thrown into the wild and not handled by the promise?

Not quite sure how you would achieve this - is it even possible to throw an exception and not have it bubble when inside a promise chain?

Yeah, don't think theres a way to distinguish between the two...


make decorators async compatible

Pull Request Test Coverage Report for Build 318

52 of 52 (100.0%)  changed or added relevant lines in 5 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 316:
0.0%

Covered Lines:
1405

Relevant Lines:
1405

💛  - Coveralls


Using Only LongNet

Hi, Thanks a lot for your work.
I would like to know if it would be possible to use only the LongNet part to extract slide level features starting from tile level features extracted from other models. Looking at the code and at the paper itself, seems like it could be possible but I cannot find a way to integrate that.

Hi, this is an interesting question. The LongNet model is built upon the tile embeddings produced by GigaPath's tile encoder. I'm not sure if it works by inputting other model's embeddings. But definitely worth to explore! Our code should be agnostic to the tile encoder, as long as the tile encoder's dimension is the same as us, e.g., 1024.


Update primary components

Contributor Comments
This updates the primary components of easy_infra.
Pull Request Checklist
Thank you for submitting a contribution to easy_infra!
In order to streamline the review of your contribution we ask that you review and comply with the below requirements:

[X] Rebase your branch against the latest commit of the target branch
[X] If you are adding a dependency, please explain how it was chosen
[X] If manual testing is needed in order to validate the changes, provide a testing plan and the expected results
[X] If there is an issue associated with your Pull Request, align your PR
title with this documentation

I get a parsing error when I pull this into a local branch and build with pipenv run invoke build,
Then run command docker run -v $(pwd):/iac seiso/easy_infra terraform plan to check the contents of the easy_infra folder or cd into ~/src/seiso/easy_infra/tests/terraform
Output:
docker run -v $(pwd):/iac seiso/easy_infra terraform plan
INFORMATIONAL:  Passed terraform init -backend=false initialization
INFORMATIONAL:  Passed terraform validate validation
ERROR:  Failed checkov directory scan

       _               _
   ___| |__   ___  ___| | _______   __
  / __| '_ \ / _ \/ __| |/ / _ \ \ / /
 | (__| | | |  __/ (__|   < (_) \ V /
  \___|_| |_|\___|\___|_|\_\___/ \_/

By bridgecrew.io | version: 2.0.534
Update available 2.0.534 -> 2.0.536
Run pip3 install -U checkov to update

terraform scan results:

Passed checks: 8, Failed checks: 12, Skipped checks: 0, Parsing errors: 1

Check: CKV_AWS_88: "EC2 instance should not have public IP."
	PASSED for resource: aws_instance.web
	File: /checkov/insecure.tf:17-25

Check: CKV_AWS_46: "Ensure no hard-coded secrets exist in EC2 user data"
	PASSED for resource: aws_instance.web
	File: /checkov/insecure.tf:17-25

Check: CKV_AWS_41: "Ensure no hard coded AWS access key and secret key exists in provider"
	PASSED for resource: aws.default
	File: /checkov/insecure.tf:27-29

Check: CKV_AWS_88: "EC2 instance should not have public IP."
	PASSED for resource: aws_instance.web
	File: /terrascan/insecure.tf:17-25

Check: CKV_AWS_46: "Ensure no hard-coded secrets exist in EC2 user data"
	PASSED for resource: aws_instance.web
	File: /terrascan/insecure.tf:17-25

Check: CKV_AWS_41: "Ensure no hard coded AWS access key and secret key exists in provider"
	PASSED for resource: aws.default
	File: /terrascan/insecure.tf:27-29

Check: CKV_AWS_23: "Ensure every security groups rule has a description"
	PASSED for resource: aws_security_group_rule.example
	File: /tfsec/insecure.tf:5-14

Check: CKV_AWS_41: "Ensure no hard coded AWS access key and secret key exists in provider"
	PASSED for resource: aws.default
	File: /tfsec/insecure.tf:1-3

Check: CKV_AWS_8: "Ensure all data stored in the Launch configuration EBS is securely encrypted"
	FAILED for resource: aws_instance.web
	File: /checkov/insecure.tf:17-25

		17 | resource "aws_instance" "web" {
		18 |   # fails checkov checks CKV_AWS_8 and CKV_AWS_79
		19 |   ami           = data.aws_ami.ubuntu.id
		20 |   instance_type = "t3.micro"
		21 |
		22 |   tags = {
		23 |     Name = "HelloWorld"
		24 |   }
		25 | }

Check: CKV_AWS_79: "Ensure Instance Metadata Service Version 1 is not enabled"
	FAILED for resource: aws_instance.web
	File: /checkov/insecure.tf:17-25

		17 | resource "aws_instance" "web" {
		18 |   # fails checkov checks CKV_AWS_8 and CKV_AWS_79
		19 |   ami           = data.aws_ami.ubuntu.id
		20 |   instance_type = "t3.micro"
		21 |
		22 |   tags = {
		23 |     Name = "HelloWorld"
		24 |   }
		25 | }

Check: CKV_AWS_135: "Ensure that EC2 is EBS optimized"
	FAILED for resource: aws_instance.web
	File: /checkov/insecure.tf:17-25

		17 | resource "aws_instance" "web" {
		18 |   # fails checkov checks CKV_AWS_8 and CKV_AWS_79
		19 |   ami           = data.aws_ami.ubuntu.id
		20 |   instance_type = "t3.micro"
		21 |
		22 |   tags = {
		23 |     Name = "HelloWorld"
		24 |   }
		25 | }

Check: CKV_AWS_126: "Ensure that detailed monitoring is enabled for EC2 instances"
	FAILED for resource: aws_instance.web
	File: /checkov/insecure.tf:17-25

		17 | resource "aws_instance" "web" {
		18 |   # fails checkov checks CKV_AWS_8 and CKV_AWS_79
		19 |   ami           = data.aws_ami.ubuntu.id
		20 |   instance_type = "t3.micro"
		21 |
		22 |   tags = {
		23 |     Name = "HelloWorld"
		24 |   }
		25 | }

Check: CKV_AWS_2: "Ensure ALB protocol is HTTPS"
	FAILED for resource: aws_alb_listener.invalid-alb-listener
	File: /invalid/invalid.tf:1-4

		1 | resource "aws_alb_listener" "invalid-alb-listener"{
		2 |     # invalid due to missing required arguments
		3 |     port     = "65535"
		4 | }

Check: CKV_AWS_103: "Ensure that load balancer is using TLS 1.2"
	FAILED for resource: aws_alb_listener.invalid-alb-listener
	File: /invalid/invalid.tf:1-4

		1 | resource "aws_alb_listener" "invalid-alb-listener"{
		2 |     # invalid due to missing required arguments
		3 |     port     = "65535"
		4 | }

Check: CKV_AWS_8: "Ensure all data stored in the Launch configuration EBS is securely encrypted"
	FAILED for resource: aws_instance.web
	File: /terrascan/insecure.tf:17-25

		17 | resource "aws_instance" "web" {
		18 |   # fails terrascan scan due to IMDS config and lack of a vpc
		19 |   ami           = data.aws_ami.ubuntu.id
		20 |   instance_type = "t3.micro"
		21 |
		22 |   tags = {
		23 |     Name = "HelloWorld"
		24 |   }
		25 | }

Check: CKV_AWS_79: "Ensure Instance Metadata Service Version 1 is not enabled"
	FAILED for resource: aws_instance.web
	File: /terrascan/insecure.tf:17-25

		17 | resource "aws_instance" "web" {
		18 |   # fails terrascan scan due to IMDS config and lack of a vpc
		19 |   ami           = data.aws_ami.ubuntu.id
		20 |   instance_type = "t3.micro"
		21 |
		22 |   tags = {
		23 |     Name = "HelloWorld"
		24 |   }
		25 | }

Check: CKV_AWS_135: "Ensure that EC2 is EBS optimized"
	FAILED for resource: aws_instance.web
	File: /terrascan/insecure.tf:17-25

		17 | resource "aws_instance" "web" {
		18 |   # fails terrascan scan due to IMDS config and lack of a vpc
		19 |   ami           = data.aws_ami.ubuntu.id
		20 |   instance_type = "t3.micro"
		21 |
		22 |   tags = {
		23 |     Name = "HelloWorld"
		24 |   }
		25 | }

Check: CKV_AWS_126: "Ensure that detailed monitoring is enabled for EC2 instances"
	FAILED for resource: aws_instance.web
	File: /terrascan/insecure.tf:17-25

		17 | resource "aws_instance" "web" {
		18 |   # fails terrascan scan due to IMDS config and lack of a vpc
		19 |   ami           = data.aws_ami.ubuntu.id
		20 |   instance_type = "t3.micro"
		21 |
		22 |   tags = {
		23 |     Name = "HelloWorld"
		24 |   }
		25 | }

Check: CKV_AWS_24: "Ensure no security groups allow ingress from <IP_ADDRESS>:0 to port 22"
	FAILED for resource: aws_security_group_rule.example
	File: /tfsec/insecure.tf:5-14

		5  | resource "aws_security_group_rule" "example" {
		6  |     # fails tfscan check AWS006
		7  |     type        = "ingress"
		8  |     cidr_blocks = ["<IP_ADDRESS>/0"]
		9  |     description = "This is an example insecure rule for tfsec testing"
		10 |     from_port   = "0"
		11 |     to_port     = "0"
		12 |     protocol    = "-1"
		13 |     security_group_id = "sg-id"
		14 | }

Check: CKV_AWS_25: "Ensure no security groups allow ingress from <IP_ADDRESS>:0 to port 3389"
	FAILED for resource: aws_security_group_rule.example
	File: /tfsec/insecure.tf:5-14

		5  | resource "aws_security_group_rule" "example" {
		6  |     # fails tfscan check AWS006
		7  |     type        = "ingress"
		8  |     cidr_blocks = ["<IP_ADDRESS>/0"]
		9  |     description = "This is an example insecure rule for tfsec testing"
		10 |     from_port   = "0"
		11 |     to_port     = "0"
		12 |     protocol    = "-1"
		13 |     security_group_id = "sg-id"
		14 | }

Error parsing file /iac/kics/insecure.tf

@callmestu by running the below command I was able to confirm that this behavior was not introduced via this PR:
docker pull seiso/easy_infra && docker run -v $(pwd):/iac seiso/easy_infra:latest terraform plan

It has the same parser error at the end.  Also, I don't believe that parser error is a true issue, as the kics/ folder is intended for kics/ tests and not checkov.


Chapter 4, PluginProcessor.cpp, change MIDI to Midi

Hi, as per coding standards, abbreviations shouldn't be TITLECASE, therefore shouldn't the method handleMIDI(...), be
void handleMidi(...) ? - Chapter 4, page. 81.
The juce library also has MIDI written as Midi in its classes, fields and methods.

Coding standards are largely a matter of opinion, and my opinion is that it's fine to write MIDI instead of Midi. 😄 But feel free to change the name to handleMidi() if you like that better!

Thanks for the reply.


Fixes to onCreateObject event handlers

In OnCreateObject handler :

changed data.unk1 and data.unk2 to data.noCameraCol and data.syncRotation
changed 65535 in attach check statements to -1
changed order of attachToVehicleId and attachToPlayerId . and change attachToPlayerId to attachToObjectId

In OnVehicleStreamIn :

change name in comment , from OnVehicleSteamIn to OnVehicleStreamIn

Thanks for contribution, but there are several issues in your changes.

noCameraCol and syncRotation should be bool8 because they are boolean
attachToObjectId goes before attachToVehicleId
signed 16-bit integers reads as unsigned values due to a bug in SAMPFUNCS, so it must stay 65535 in that id check

Corrected the mistakes . thanks


[BUG] Latest version of flutter is not installed correctly

Before creating a bug report please make check the following

[ ] You have read our FAQ
[ ] If you have used flutter. Please install correctly, run pub cache repair. Close the terminal and try again.
[ ] If you are on Windows. Make sure you are running the terminal as administrator or with developer permissions.
[ ] Run fvm doctor if possible and add the output to the issue.

Describe the bug
Run the fvm install command for flutter version 3.19.5, the command runs without error but the fvm list command shows that the version needs setup. As shown follow

To Reproduce
Steps to reproduce the behavior:

Go to terminal..
Run fvm install 3.19.5...
Command finished successfull
Run fvm list
The command shows the version 3.19.5 not ready.

Expected behavior
The version must be installed and will be available for use like other versions.
Logs
No logs
Desktop (please complete the following information):

OS: Windows
FVM Version: 3.1.3
If Windows: Which Powershell are you using? PowerShell version: 5.1.19041.4170 , CommandLine Version: 10.0.19045.4170

Additional context
No more information

the parameter --setup do the configuration


triggers.show !== 'none' in case popover-trigger="'none'" will be always "true"

Hello, found some bug in version 2.4.0. When i try set popover-trigger="'none'" i figured out comparison: triggers.show !== 'none' in case popover-trigger="'none'" will be always "true"

and it's bad
 } else {
                triggers = getTriggers(val);
              }

              **if (triggers.show !== 'none') {**
                triggers.show.forEach(function(trigger, idx) {
                  if (trigger === 'outsideClick') {
                    element.on('click', toggleTooltipBind);
                    $document.on('click', bodyHideTooltipBind);

That should be an indexOf check - want to PR the simple fix for this?

Hmm...I'm not sure what the correct check is actually, but maybe it's because I'm on 4 hours of sleep. It looks like triggers.show.indexOf('none') !== -1 should be sufficient though.

You mean: triggers.show.indexOf('none') === -1

Yeah...not my morning :( .

Hi @qwq-qwq i need to use the same test to fix #6413
Assigning a flag on ttScope if its 'none' so i can use it in there line 4936:
    function keypressListener(e) {
      if (e.which === 27) {
        var last = openedTooltips.top();
        if (last && !last.key.noneTrigger) {
          last.value.close();
          last = null;
        }
      }
    }

Sorry for awake this now but is this the same as this?
tooltip-trigger="none" is being ignored. When I pass the mouse over it still displays the tooltip.


Consistent type responses, destructuring

This PR changes the responses in certain types to be consistent with one another, fixes wording in the member type ("member" instead of "user"), and destructures some things that were not.

The "user" vs "member" wording was completely intentional, but I've reconsidered it.


Bit Events and Listen

Updating channel-bitsevents to channel-bits-events-v1
Changing how Listen works to compile a list of events to listen to before firing the entire Listen off in one request, because you're going to be specifying the same Oauth key to all of the listen requests anyways.

Commenting from my phone to let you know that I see it and wanted to thank you. I'll be able to review tomorrow or Saturday as I've been quite busy.
Cheers.

Hey, no problem.  I tried writing a full PubSub solution on my own and it gave me a wall of problems.  This has neatly done the heavy lifting I needed, and was easy to work with.
Regarding Listen, each ListenTo_ function adds to the list of topics that are combined together and sent as one request when you call sendTopics ();.

Looks good, thanks!


[stable/ambassador] Add HOST_IP environment variable to ambassador container

What this PR does / why we need it:
Add HOST_IP environment variable for Consul integration:

https://www.getambassador.io/reference/core/resolvers/#the-consul-resolver

Which issue this PR fixes
Fixes https://github.com/helm/charts/issues/13555.
Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md (N/A)
[x] title of the PR contains starts with chart name e.g. [stable/chart]

/assign @kflynn

/ok-to-test
/assign

/lgtm


[FEATURE] Convert the backend roadmap to a readme format

Description
Convert the [backend roadmap on CC website](https://www.commclassroom.org/backend-roadmap to a readme.md format in the backend folder.
Example:
#Frontend Roadmap
## Step one
- Resource 1
- Resource 2

Hey, I wanted to work on this issue.

Thanks @siddhant-khisty


opencv2/core/core.hpp: No such file or directory

Error:(2, 33) opencv2/core/core.hpp: No such file or directory

Thanks @younas-bangash! I will check soon!


ServerError (timed out) when using NuxtJS in SSG mode with Sanity

Describe the bug
I have a NuxtJS (Version 2) app running in SSG mode and Sanity.io as CMS. NuxtJS generates/prerenders about 300 pages (data comes from Sanity API), which takes quite some time. The problem is that after a while I get the following error (at build stage):
ServerError: The operation timed out
So I guess Sanity is closing the server connection, while NuxtJS is still generating pages. I reduced the amount of pages to about 280 and then it works. So it seems like Sanity closes the connection too early.
Possible solutions

Is there a way to keep the server connection to Sanity.io longer alive, before the operation is timed out?
Is there a way to split up the static site generation with NuxtJS, so that after (let‘s say) 50 pages the current connection to sanity closes and a new one starts for the next 50 pages?

Is there any other solution you can think of?

If you have a shared body of information you request on each page, consider https://nuxtjs.org/docs/configuration-glossary/configuration-generate/#speeding-up-dynamic-route-generation-with-payload.
Otherwise, you can adjust generate.concurrency and/or generate.interval.


Bug introduced from 1.1.4 and later

Subject of the issue
My unit tests suddenly started breaking from 1.1.4 and later
Steps to reproduce
since updating to 1.1.4 (or greater) my unit tests are now breaking - I have a custom component (renderer) which renders form components (e.g. input, text area, checkbox etc) dynamically using a config provided via props. The component iterates through the config and renders the inputs (mapping them to appropriate vue components) - now in my test when i do the following:
...
const dynamicFormConfig = ['text-area', 'text-input'];

const options = merge({
    propsData: {
      dynamicFormConfig,
    },
    store,
    localVue,
  }, params.options);

textInput = mount(RendererComponent, options).findComponent({` name: 'MyTextInput' });

expect(textInput.exists()).toBe(true);

i should get true, but now my tests fail with false.
Expected behaviour
I should get true from the test expectation but now im getting false. When i downgrade to 1.1.3 everything works fine and the tests pass.
Actual behaviour
The test now fails. (i.e. textInput.exist() returns false)
Possible Solution
Revert the breaking change introduced in 1.1.4 (Djaler:fix/find-functional-component)

I don't think we should revert a change that fixes a real bug. Let's just fix this new bug. Can you please post your entire test and component? We can add it to the test suite, then make it pass. Then both bugs will be fixed. What do you think? 👍

ok sure i'll put a demo repo together which replicates this problem

Thank you!

I've worked with ravjsdev to create a minimal example which reproduces the issue. Here it is: https://github.com/GeorgieH/vue-test-utils-1845
Info is in the README. Let me know if you have any questions.

Thanks.

Released in 1.2.1 https://github.com/vuejs/vue-test-utils/releases/tag/v1.2.1
This should be good now.


[brightcove] 'programmedContent' is None

$ python -m youtube_dl --verbose 'http://www.11alive.com/videos/news/local/2015/10/27/74703190/'
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'http://www.11alive.com/videos/news/local/2015/10/27/74703190/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.11.02
[debug] Git HEAD: a230068
[debug] Python version 2.7.10 - Darwin-15.0.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 2.8.1, ffprobe 2.8.1
[debug] Proxy map: {}
[generic] 74703190: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 74703190: Downloading webpage
[generic] 74703190: Extracting information
[generic] Brightcove video detected.
[download] Downloading playlist: VIDEO: All Three Angles of Spring Valley Incident
[generic] playlist VIDEO: All Three Angles of Spring Valley Incident: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[Brightcove]<PHONE_NUMBER>001: Downloading webpage
[Brightcove]<PHONE_NUMBER>001: Extracting information
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/pdm/Source/youtube-dl/youtube_dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "youtube_dl/__init__.py", line 410, in main
    _real_main(argv)
  File "youtube_dl/__init__.py", line 400, in _real_main
    retcode = ydl.download(all_urls)
  File "youtube_dl/YoutubeDL.py", line 1666, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "youtube_dl/YoutubeDL.py", line 672, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "youtube_dl/YoutubeDL.py", line 833, in process_ie_result
    extra_info=extra)
  File "youtube_dl/YoutubeDL.py", line 725, in process_ie_result
    extra_info=extra_info)
  File "youtube_dl/YoutubeDL.py", line 661, in extract_info
    ie_result = ie.extract(url)
  File "youtube_dl/extractor/common.py", line 290, in extract
    return self._real_extract(url)
  File "youtube_dl/extractor/brightcove.py", line 237, in _real_extract
    videoPlayer[0], query_str, query, referer=referer)
  File "youtube_dl/extractor/brightcove.py", line 266, in _get_video_info
    video_info = info['programmedContent']['videoPlayer']['mediaDTO']
TypeError: 'NoneType' object has no attribute '__getitem__'

The Brightcove URL extractor looks at the  tag at the top of the page and finds the URL
http://c.brightcove.com/services/viewer/htmlFederated?isVid=1&isUI=1&playerID=2857340256001&autoStart=true&%40videoPlayer=4582511272001

That page is downloaded and its info extracted but it's missing a programmedContent section, which we use to get the video URL from.
If we change the videoPlayer parameter in the above URL from<PHONE_NUMBER>001 to ref%3A74703190 (which is the basename of the URL given on the command line) then we get a valid programmedContent object.

Possible duplicate of https://github.com/rg3/youtube-dl/issues/3139

The link provided is dead
'http://www.11alive.com/videos/news/local/2015/10/27/74703190/'
here is an current example link
'https://www.11alive.com/video/news/local/food-bank-in-atlanta-seeing-increase-in-need-how-to-help/85-8c9c321a-f68f-40f5-90b5-a5740997e5d3'
youtube-dl failed to download videos from 11alive.com
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.12.2 (CPython) - macOS-14.5-arm64-arm-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[generic] 85-8c9c321a-f68f-40f5-90b5-a5740997e5d3: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 85-8c9c321a-f68f-40f5-90b5-a5740997e5d3: Downloading webpage
[generic] 85-8c9c321a-f68f-40f5-90b5-a5740997e5d3: Extracting information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'https://www.11alive.com/video/news/local/food-bank-in-atlanta-seeing-increase-in-need-how-to-help/85-8c9c321a-f68f-40f5-90b5-a5740997e5d3'
[download] Destination: Food bank in Atlanta seeing increase in need _ How to help -85-8c9c321a-f68f-40f5-90b5-a5740997e5d3.unknown_video
[download] 100% of 202.82KiB in 00:00

Hi @yinshiyi, since youtube-dl is relatively dead, it might be better for us to move to https://github.com/yt-dlp/yt-dlp now.

This is just a really antique issue that no-one ever closed. If the issue persists, especially with the latest nightly build, any concerned user is invited to open a new issue in the usual way. Thx @yinshiyi for noticing this.


Update LlbcAstUtils.chain_statements

This is a very minor cleanup.

I don't understand the CI failing on Kyber: I think the failure doesn't come from changes on our side.


End device(Light) is not shown on mobile application

https://github.com/google-home/sample-app-for-matter-android/issues/114

Thanks for submitting this fix, much appreciated! If you could please fill in the CLA, then the PR can be merged.

pierredelisle: I have completed CLA. Please let me know if any.


Extension is removing QuickFixes from other extensions.

When using this and eslint, VSCode no longer can load any "Quick Fixes".
Disabling this extension restores the Quick Fixes from eslint.
When it fails it seems to try for a very, very long time to resolve the "Quick Actions".
The only error I could find in the logs was this (not sure it's relevant though).
2023-10-25 12:51:57.665 [warning] AndreasArvidsson.andreas-talon - Code actions of kind 'quickfix 'requested but returned code action is of kind 'refactor.rewrite'. Code action will be dropped. Please check 'CodeActionContext.only' to only return requested code actions.

Oops, this looks like it's unrelated and was just a random coincidence that disabling the extension got it working again.


Json codable no type assumption in custom fromJson

fix https://github.com/dart-lang/sdk/issues/59580
related: https://github.com/dart-lang/sdk/issues/55989

json_codable wrongly assumes fromJson receives a Map

This allows an user defined class to have a fromJson method with a parameter that is not a Map. Before this change the @JsonCodable macro would check if a field has a fromJson constructor and assume that the parameter type was a Map. This PR uses the type of the parameter instead.
Note: I have this weird behavior where I can only run the tests with the macros enabled once per branch, so I have to switch branches and run the tests on another.

@jakemac53 not sure if updates to this package are welcome yet or not. I've another down the drain for enums but I cannot make it work currently because the behavior is different during test than in the augmented code which is puzzling

It seems there is already a PR for this here : https://github.com/dart-lang/sdk/pull/56665

Can a class also define how to handle null? Would factory Foo.fromJson(Object? o) or factory Foo.fromJson(String? s) work? Can we add a test if so?

Can a class also define how to handle null? Would factory Foo.fromJson(Object? o) or factory Foo.fromJson(String? s) work? Can we add a test if so?

It does not currently handle that case no

@natebosch I added a test for String? o

This is released now 👍


Error: App Not Setup. This app is still in development mode

Hi,
Thank you for this great script.
When I try to authorize my account to get my access token I get this error:
Error: App Not Setup. This app is still in development mode, and you don't have access to it.....
How to solve this issue?

Thank you very much


android sdk 31, android exported fix

Fixes the issue https://github.com/mauron85/react-native-background-geolocation/issues/571.
The android play store doesn't accept the app bundle otherwise

Merge

Can you help me to resolve this?
FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file<EMAIL_ADDRESS>line: 29

* What went wrong:
A problem occurred evaluating project ':mauron85_react-native-background-geolocation'.
> Could not read script<EMAIL_ADDRESS>as it does not exist.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
A problem occurred configuring project ':mauron85_react-native-background-geolocation'.
> compileSdkVersion is not specified. Please add it to build.gradle


METRON-671: Refactor existing Ansible deployment to use Ambari MPack

Updated Update - I think this is ready for review.
Update - Documentation pending, this is pretty much ready to go and could use some eyes on. The major change is that you'll need Docker (Docker for Mac on Macs) running for the build to complete. This is because the MPack requires the RPMs be built.
To test, run Quick Dev or Full Dev or both.
I'll be working on the documentation in the next day or so.
This is the first set of changes to enable Ansible installation to use the Ambari MPack. It currently works (in my environment) with full-dev using sensor-stubs. It will not work with Quick-Dev or EC2 at this time.
It's (well past - sorry) starting to get large, so I wanted to push it out for feedback while I'm working issues with the distributed install.
Some points of interest:

As discussed, components (ES, Kibana, Metron Topologies) that are managed by Ambari are removed from Monit
I had added a notion of 'required configurations' to the blueprint and ambari_cluster_state script so blueprint verification would pass.
I've refactored our Ansible host groupings (see inventory/full-dev-platform/hosts). The Metron group is now the host that we use as the Metron services master.
I cleaned up the very minimum number of issues that @mattf-horton found in METRON-609 required to make single-node deployment work with the MPack. It's my intention to work to get his changes incorporated on top of the final version of this changeset.
I removed the Solr role from Ansible. My current intention is to open a Jira to add a Solr Indexing component to the MPack.

Immediate next steps:

[X] Get Full-Dev working.
[X] Get the EC2 distributed install working.
[X] Get Quick-Dev with enrichment replacement working.
[X] Adjust readmes and docs where required.
[X] Document test plan

I also hope to receive and respond to feedback.

How are you running the playbook?  I cannot get it to execute

I am, for both full dev and ec2. What are you trying to do?

I want to run the playbook to just build the rpm's, not deploy.  So just metron_build

Yes, that works on my rig. Can you tell me a bit more about what you're experiencing?

can you share your command line?    Did you copy or create an ansible.cfg?

Sure. It's vagrant up or ./run.sh. :)  I run it as part of running full dev or ec2.
What error are you getting?
Btw, to only build the rpms, you may can still do an mvn package -DskipTests -Pbuild-rpms from maven-deploy.

I need to run the playbook though.
It doesn't match any hosts

ansible-playbook -v playbooks/metron_build.yml
Using /Users/ottofowler/src/apache/forks/incubator-metron/metron-deployment/ansible.cfg as config file
[WARNING]: provided hosts list is empty, only localhost is available

< PLAY >
    \   ^__^
     \  (oo)\_______
        (__)\       )\/\
            ||----w |
            ||     ||

skipping: no hosts matched

< PLAY RECAP >
    \   ^__^
     \  (oo)\_______
        (__)\       )\/\
            ||----w |
            ||     ||

Do I have to create an inventory with my local machine name?

Yeah, that seems like a good thing to try.

ansible-playbook -v -i "localhost," -c local playbooks/metron_build.yml
now it is running and i'll see about the errors

Maybe we should add that line to the doc or create a script??

No. It's not meant for that. If you want to build standalone, Maven is the supported way.

ok - when you review what I'm doing in the playbook/role we can talk about alternatives

Sounds good. The important thing to remember is the overall goal of reducing reliance on Ansible. I had to add a build task because installation will fail without it, but I fully expect that task to disappear sooner rather than later, so adding additional dependencies on it will need to be scrutinized carefully.

Well, if we don't want to put deployment things with the src, then ansible is a more flexible and easier to use tool for certain tasks too. But we will talk about it when I get it working

David, after fetch and merge of your latest I can no longer vagrant up full_dev
==> node1: Adding box 'new_base' (v0) for provider: virtualbox
node1: Downloading: /Users/dml/projects/metron-dlyle/metron-deployment/packer-build/builds/base-centos-6.7-2.1.20170303223924.git.33abe8cf13c347a2dfdece145a7b8c17f2a423c0_dirty.virtualbox.box
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try
again.
Couldn't open file /Users/dml/projects/metron-dlyle/metron-deployment/packer-build/builds/base-centos-6.7-2.1.20170303223924.git.33abe8cf13c347a2dfdece145a7b8c17f2a423c0_dirty.virtualbox.box
➜  full-dev-platform git:(METRON-258-RPM)

@ottobackwards - that's correct. At that time the new boxes hadn't been prepped. Sorry for the confusion, I had left that box uncheck in an effort to communicate that. It should be good now.

This is ready for review. Thanks!

@dlyle65535 can you point to any documentation for the ambari api/functions that we are using?

@ottobackwards - sure, I used this as a guide.

From @JonZeolla - "Looks like Docker is a dependency now which rules out CentOS 6 if we strictly follow the docker documentation.  Was that expected or would you assume people would opt for the workaround steps?"
It is, to integrate with the MPack we have to build RPMs and (right now anyway), so that means Docker. It was always implicit in this ticket, but I did try to underline it over the weekend. I wasn't aware that Docker and CentOS6 were problematic. I used to use those two together a little while back, but it was forever ago in Dockertime.
Is that a showstopper for anyone?

Not a showstopper for me, but maybe worth a mention in Upgrading.md that CentOS 6 shouldn't be used any longer.  Note that Upgrading.md gets a bit of a modification in METRON-752 (#470).

@JonZeolla - makes sense, I've pushed that change.

I'm working on those merge conflicts right now.

Resolved the merge conflicts from Master and addressed review comments.

@justinleet - wrt the Dashboard changes, the map shows up. If you've saved off a bunch of changes, you may want to clobber mine, but make sure the map shows up when you're done.

Something bad happened when I tried to run "Quick Dev" with the latest code.

I ran a vagrant up and the command completed successfully.
But no "deployment report" was shown.
I cannot access Ambari on 8080, even though ambari-server is running.

See selected output from the attempt here.  It seems like we're not matching to hosts/host groups correctly after the sensor deployment.
On a side note, prior to the latest commits, yesterday I was able to run "Full Dev" successfully.  I did not have a chance to try "Quick Dev" before the latest commits to know if this is something that was newly introduced.

@justinleet - wrt the Dashboard changes, the map shows up. If you've saved off a bunch of changes, you may want to clobber mine, but make sure the map shows up when you're done.

I was going to ask the same question.  So did the dashboard need to change?  Seems like you are saying it did not.  If not, then let's take those changes out of the PR.

@nickwallen - it did need to change. The map didn't work. It does now. If @justinleet finds it easier to duplicate that change in his work, he can do so, otherwise, he should build on the dashboard.p file in this PR as to not undo what I've done.
wrt the failure, it's strange- Ansible agrees with you that Ambari is running. I'll spin it up and see what I can see. Can you curl to Ambari?

@dlyle65535 @nickwallen I'm not sure what the exact fix was with the dashboard.  Either approach to fixing will work for me, but I don't know what made the map not work to fix it?

but I don't know what made the map not work to fix it?

Technically, we had two versions of the dashboard.  Maybe the "legacy" dashboard deployed via Ansible using elasticdump was always working, but the map on the dashboard deployed via the MPack had been broken prior to Dave's work on this PR.  Just a guess.  I would like to know the "why" too.

Chill guys, I'm trying to get a screenshot. ;)

There's a checkbox in the widget config:

That needs to be checked and the rest auto-populated:

I was able to start over on "Quick Dev" and get it running.  Maybe I wasn't patient enough on the first run.
It still did not show the deployment report, so I'm not sure if it is only missing that step or others.

@nickwallen - glad of that, I did see that I was still using a stale inventory for quick-dev which worked, but without the report. I've got a fix for that in test now.

@dlyle65535 Perfect, thanks.  I'm going to go ahead and just make that change and test in 745.  If this goes in first, 745 just takes its dashboard changes entirely.  In the less likely event that 745 goes in first, this PR just accepts the changes entirely.

I can offer a slightly different perspective on this PR, from the pov of someone working downstream ( i rebased on it ), and who has been working on adding things to the spec and install.
Many of my issues have been of my own doing ( trying and failing to use templates for spec's but not go all the way ), not doing everything in maven from the start since ansible is more flexible etc .
What I would like to bring up is the 'black box' nature of this build now.  I don't see errors surface into the ansible log they way they would before.
The build starts building, and while it is running all the maven commands there is no output, and no logging.   I have had to drop down into the rpm level and build there to see errors, or go into the vagrant image and hunt down logs to see problems with deployment ( a good tip is when the ansible log says it is deploying cluster, connect to ambari and monitor the ops, which tail the logs and give the file names, with are NOT var/log based ).
The point of this specific feedback is that we have added a very large amount of time to the vagrant up, during which many many things could go wrong, and none of them surface to the user.

That's good feedback @ottobackwards. Post build errors will now surface in Ambari. I don't know what to do about the build. Any errors would eventually pop up on your screen. Is there any visual cue that would help?

@dlyle65535 @ottobackwards Can we link the various Ambari logs so they're visible on the local machine?  Anything that blows up in the UI should blow up in the logs, which means everything is searchable like it was before.

If someone is doing what the old workflow is ( just from the command line - not opening ambari until everything is done ) they will never see anything but the fail, please fix and try again.  I don't think we should have this be a 'two parter', where the caller has to go hunting through different things in the vm and out of it to hunt things down.   The cadillac fix would be for a post build step that gathers the logs into a buildResult dir or something.  The maven output that is 'eaten', the *.txt files that the ambari ui refers to.
Also, if someone wanted to use automation with this, it will be a problem.
There is, to me, a change with this, from a modular build where we were doing multiple build steps, with each in and of itself being self documenting, to a monolithic build

@ottobackwards - I don't agree that automation breaks with this. It is different though. If you're building automation now, you'd probably want to leverage Ambari Blueprints.
Unfortunately, as we go down this path, I don't see a way around checking Ambari. In reality, that was always the case. If the Hadoop deployment failed, that was where you looked. If the probes failed to start, you checked Monit. Ansible alerted you to the fact of the failure, but anything but the simplest required poking around Ambari/Monit/Command Line.
It sounds to me like you'd like to revisit the decision to reduce reliance on Ansible. If that's the case, we can certainly do that. I thought we had consensus but maybe we don't. Maybe we need to go back to the discuss thread?

@nickwallen - last commit fixed deployment report for Quick Dev. Thanks!

No, I don't have a problem with the ansible decision.  And I don't even mean for this feedback to hold up this PR.  I just think it is harder to troubleshoot and work with it since you don't see logs for a substantial portion of the process.  Since there isn't another person looking at developing on top of this vs. the resulting functionality, I wanted to bring that up.   We can log jiras to make it better or whatever.

@ottobackwards - thanks for that! Let's do make sure the end result is something you can use. Also remember that this is steps 1 and 2 of an at least 4 step improvement, so perhaps once we get this to a working state, it be a good time to re-check the underlying assumptions to make sure we're still on a working path?

you point on blueprints reminds me that this kind of obsoletes the 'deploying metron on an ambari managed cluster' wiki page, and that page is very valuable

I'm fine with that.  Just being up front on the change is enough from now.  Troubleshooting this may be a big user issue generator

I have been able to launch "Quick Dev" with deployment report.  Thanks for the fix @dlyle65535
I have been fighting a bit with the AWS deployment.  I ran into two issues.
(1)   On one pass the setup of Ambari seems to fail, but the deployment continued, which causes it to fail later on in the deployment.  To fix, I manually logged into the host and ran the Ambari setup and then re-ran the deployment which addressed the problem.
I am almost certain that I have seen this before prior to the work in this PR.
$ ./run.sh
...

TASK [ambari_master : Setup ambari server] *************************************
...

"Successfully downloaded JDK distribution to /var/lib/ambari-server/resources/jdk-8u77-linux-x64.tar.gz", "Installing JDK to /usr/jdk64/", "Successfully installed JDK to /usr/jdk64/", "Downloading JCE Policy archive from http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip to /var/lib/ambari-server/resources/jce_policy-8.zip", "", "Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/jce_policy-8.zip", "Installing JCE policy...", "Completing setup...", "Configuring database...", "Enter advanced database configuration [y/n] (n)? ", "Configuring database...", "Default properties detected. Using built-in database.", "Configuring ambari database...", "Checking PostgreSQL...", "Running initdb: This may take up to a minute.", "Initializing database: [  OK  ]", "", "About to start PostgreSQL", "Configuring local database...", "Connecting to local database...connection timed out...retrying (1)", "Connecting to local database...connection timed out...retrying (2)", "Connecting to local database...unable to connect to database", "ERROR: could not change directory to \"/home/centos\"", "psql: FATAL:  the database system is starting up", "", "ERROR: Exiting with exit code 2. ", "REASON: Running database init script failed. Exiting."], "warnings": []}

$ ./run.sh
...

TASK [ambari_config : check if ambari-server is up on ec2-52-37-229-181.us-west-2.compute.amazonaws.com:8080] ***
fatal: [ec2-52-37-229-181.us-west-2.compute.amazonaws.com]: FAILED! => {"changed": false, "elapsed": 300, "failed": true, "msg": "Timeout when waiting for ec2-52-37-229-181.us-west-2.compute.amazonaws.com:8080"}

(2) The second issue was more unexpected.  On all but one of the 10 AWS nodes, the deployment went smoothly.  At some point during the deployment, Ansible could not talk to one node, but it continued on anyways.  After the 9 were finished, Ambari showed all 10 nodes, except the one, which it showed in yellow indicating that it could not get a heartbeat.
After Ansible was done with the 9 nodes, it then seemed to almost start over on the last node.  It went and rebuilt the source code, pushed out the RPMs, reinstalled the MPack, etc.  That really confused the cluster and it has not processed any data.
I'm sure a little manual effort could fix-up the cluster, but the behavior of Ansible was weird.  Before when I've worked with the AWS deployment, it would fail if any one node failed.  Now it seems to retry failed nodes at a later point in time, which has some negative implications when we expect actions like the build, mpack install, etc to only occur once.
Not sure what to make of this issue.

Travis failure. Upsetting. This came in with my latest Master merge. Don't fail locally with Travis command line. Log below. It's in unrelated code. Any ideas?

T E S T S
Running org.apache.metron.common.stellar.maas.StellarMaaSIntegrationTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.359 sec - in org.apache.metron.common.stellar.maas.StellarMaaSIntegrationTest
Running org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolverIntegrationTest
2017-03-07 21:54:03 ERROR ServiceDiscoverer:171 - instance must be started before calling this method
java.lang.IllegalStateException: instance must be started before calling this method
at com.google.common.base.Preconditions.checkState(Preconditions.java:176)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getChildren(CuratorFrameworkImpl.java:379)
at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.queryForNames(ServiceDiscoveryImpl.java:276)
at org.apache.metron.maas.discovery.ServiceDiscoverer.updateState(ServiceDiscoverer.java:130)
at org.apache.metron.maas.discovery.ServiceDiscoverer.lambda$new$0(ServiceDiscoverer.java:94)
at org.apache.metron.maas.discovery.ServiceDiscoverer$$Lambda$4/1527152775.childEvent(Unknown Source)
at org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:685)
at org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:679)
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:84)
at org.apache.curator.framework.recipes.cache.TreeCache.callListeners(TreeCache.java:678)
at org.apache.curator.framework.recipes.cache.TreeCache.access$1400(TreeCache.java:69)
at org.apache.curator.framework.recipes.cache.TreeCache$4.run(TreeCache.java:790)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Formatting using clusterid: testClusterID
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 1.249 sec <<< FAILURE! - in org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolverIntegrationTest
org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolverIntegrationTest  Time elapsed: 1.249 sec  <<< ERROR!
java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.server.AuthenticationFilter.constructSecretProvider(Ljavax/servlet/ServletContext;Ljava/util/Properties;Z)Lorg/apache/hadoop/security/authentication/util/SignerSecretProvider;
at org.apache.hadoop.http.HttpServer2.constructSecretProvider(HttpServer2.java:447)
at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:340)
at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:114)
at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:290)
at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:126)
at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:752)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:638)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:811)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:795)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1111)
at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:982)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:811)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:471)
at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:430)
at org.apache.metron.integration.components.MRComponent.start(MRComponent.java:58)
at org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolverIntegrationTest.setup(ClasspathFunctionResolverIntegrationTest.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolverIntegrationTest  Time elapsed: 1.249 sec  <<< ERROR!
java.lang.NullPointerException
at org.apache.metron.integration.components.MRComponent.stop(MRComponent.java:66)
at org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolverIntegrationTest.teardown(ClasspathFunctionResolverIntegrationTest.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Running org.apache.metron.common.cli.ConfigurationManagerIntegrationTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.251 sec - in org.apache.metron.common.cli.ConfigurationManagerIntegrationTest
Results :
Tests in error:
org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolverIntegrationTest.org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolverIntegrationTest
Run 1: ClasspathFunctionResolverIntegrationTest.setup:49 » NoSuchMethod org.apache.ha...
Run 2: ClasspathFunctionResolverIntegrationTest.teardown:63 » NullPointer
Tests run: 8, Failures: 0, Errors: 1, Skipped: 0

@nickwallen - I had run EC2 testing a bunch and it worked post at least as well as it did prior (sometimes AWS zigs when it should zag). But, I have made a quite few changes since my last EC2 run. I'll spin it up and see if I get it too.

@nickwallen - For your first issue, I think we're hitting a transient issue with Ambari where ambari-setup -s completes successfully but Ambari won't actually start. I'll see if I can get some better diagnosis.
On the second, did Ambari report that the install succeeded? Or did it fail and I didn't catch it?
In other news- good news, bad news. Good news: I am able to replicate the integration test failure by running them in my local environment. Bad news: it's not in the code I touched and I'm completely flummoxed. Help would be much appreciated. Once I can get past these EC2 issues, I can diff master, but like I said. Help? Appreciated.

Good news, I think I found the issue with the failing tests. The Maven reported "duplicated" dependencies weren't. I've replaced them. Travis will tell.
@nickwallen - I did see the error you're talking about in your first point above. I think your memory is correct, it's one of those transients that we see sometimes. There's not much that can be done, but I  am testing a patch that looks for "FATAL" in the ambari-setup stdout so at least we'll fail where the problem occurs.

@nickwallen - I pushed up a changeset that will address both your points. For 1, I added a test to ambari-setup to fail if FATAL appears in stdout. Making sure the EC2 build doesn't run the quick_dev role addresses the second.

@justinleet - that last commit adds the quotation requiremnt to the tool tip.

@dlyle65535 METRON-745 is in (as I'm sure you can tell from the conflict list).  I already incorporated the Kibana map changes, so you should just be able to accept master's version.

@justinleet - Accepted. Thanks.

@dlyle65535 fyi I'm deploying to ec2 right now. I'll update shortly.

@dlyle65535 is the test for EC2 to just verify everything spins up as normal? Any additional specific items to test or smoke test?

@mmiklavc - Exactly. We were wanting to make sure that @nickwallen was having environmental issues.

Looks like everything came up correctly for me in AWS. So +1 to that part of it.

Thanks @mmiklavc!

That's great, @mmiklavc.  I am also a +1.  I was able to test this successfully on Quick Dev and Full Dev.

Thanks for all the help! I intend to merge this in tomorrow afternoon. @ottobackwards, @justinleet, I wanted to make sure you two were good. I also want to make sure there's no other feedback I've missed.

I am +1.  I have been working downstream and building off of this for a bit, and been able to get quick and full up with everything started.  I have some things that I would like to improve on, but like @dlyle65535 says, this is step 2 of many

I'm +1.  I was just waiting for the EC2 component, but was able to get quick-dev, etc. spun up without issue.


[FEATURE REQUEST] Like, Save and Share options for each portfolio

Is your feature request related to a problem? Please describe.
No
Describe the solution you'd like
Implement Like, Save and Share options for each portfolio cards, display the total number of likes each portfolio got (should be dynamic and live across the site). The database setup(supabase) and auth pages are already implemented.
Additional context
The portfolio card should look like

Hey @Kiran1689 , I'd love to work on this issue. Is it also possible to break this down into three issues each for like, save and share?

Hey @Kiran1689 , I'd love to work on this issue. Is it also possible to break this down into three issues each for like, save and share?

@KartikChinda Please create new separate issues and mention this issue in them.

@Kiran1689 Hello there! Just wondering what's the status of this feature request and maybe I could take over one of the subissues if that is okay with everyone?

@blacksheepIV Go ahead!!


mtprotoproxy v1.0.5 is not compatible with Telegram Android client v 5.10

I've encountered an issue with mtprotoproxy v1.0.5 after updating Telegram Android client to version 5.10.
Telegram client was constantly reconnecting to the proxy.
Updating mtprotoproxy to the latest git master has solved the issue.
It seems to be related with the replay attack protection.
I strongly advise to make a new release.

Update.. it may be not related with the replay attack protection because official c++ mtproxy doesn't work too.
I don't know the reason, but:
1 ) official c++ mtproxy doesn't work
2) mtprotoproxy v1.0.5 doesn't work
3) mtprotoproxy master branch works fine

The problem was caused by a broken Telegram Android client. Probably there was a build with a bug. Latest official build works fine now.


Periodically get a cgi.FieldStorage instead of a string on uploads

For some reason, and I'm not sure why it happens, sometimes the POST dictionary contains cgi.FieldStorage classes instead of strings (I think). We should figure out what cases this happens in, and guard against them however we can.

https://sentry.io/python-software-foundation/warehouse-production/issues/187224559/
ProgrammingError: can't adapt type 'cgi_FieldStorage'
  File "sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)

ProgrammingError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (psycopg2.ProgrammingError) can't adapt type 'cgi_FieldStorage' [SQL: 'INSERT INTO releases (name, version, author, author_email, maintainer, maintainer_email, home_page, license, summary, description, keywords, platform, download_url, _pypi_ordering, _pypi_hidden, cheesecake_installability_id, cheesecake_documentation_id, cheesecake_code_kwalitee_id, requires_python, description_from_readme) VALUES (%(name)s, %(version)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(home_page)s, %(license)s, %(summary)s, %(description)s, %(keywords)s, %(platform)s, %(download_url)s, %(_pypi_ordering)s, %(_pypi_hidden)s, %(cheesecake_installability_id)s, %(cheesecake_documentation_id)s, %(cheesecake_code_kwalitee_id)s, %(requires_python)s, %(description_from_readme)s)'] [parameters: {'name': 'mdsplus', 'version': '7.7.6', 'author': 'Tom Fredian,Josh Stillerman,Gabriele Manduchi', 'author_email'<EMAIL_ADDRESS>'maintainer': '', 'maintainer_email': '', 'home_page': 'http://www.mdsplus.org/', 'license': '', 'summary': 'MDSplus Python Objects', 'description': '\n      This module provides all of the functionality of MDSplus TDI natively in python.\n      All of the MDSplus data types such as signal are represented as python classes.\n      ', 'keywords': FieldStorage('keywords', 'physics'), 'platform': '', 'download_url': 'http://www.mdsplus.org/mdsplus_download/python', '_pypi_ordering': None, '_pypi_hidden': False, 'cheesecake_installability_id': None, 'cheesecake_documentation_id': None, 'cheesecake_code_kwalitee_id': None, 'requires_python': '', 'description_from_readme': None}]
(51 additional frame(s) were not displayed)
...
  File "warehouse/raven.py", line 41, in raven_tween
    return handler(request)
  File "warehouse/cache/http.py", line 69, in conditional_http_tween
    response = handler(request)
  File "warehouse/cache/http.py", line 33, in wrapped
    return view(context, request)
  File "warehouse/csrf.py", line 38, in wrapped
    return view(context, request)
  File "warehouse/forklift/legacy.py", line 791, in file_upload
    .filter(Release.project == project)

ProgrammingError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (psycopg2.ProgrammingError) can't adapt type 'cgi_FieldStorage' [SQL: 'INSERT INTO releases (name, version, author, author_email, maintainer, maintainer_email, home_page, license, summary, description, keywords, platform, download_url, _pypi_ordering, _pypi_hidden, cheesecake_installability_id, cheesecake_documentation_id, cheesecake_code_kwalitee_id, requires_python, description_from_readme) VALUES (%(name)s, %(version)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(home_page)s, %(license)s, %(summary)s, %(description)s, %(keywords)s, %(platform)s, %(download_url)s, %(_pypi_ordering)s, %(_pypi_hidden)s, %(cheesecake_installability_id)s, %(cheesecake_documentation_id)s, %(cheesecake_code_kwalitee_id)s, %(requires_python)s, %(description_from_readme)s)'] [parameters: {'name': 'mdsplus', 'version': '7.7.6', 'author': 'Tom Fredian,Josh Stillerman,Gabriele Manduchi', 'author_email'<EMAIL_ADDRESS>'maintainer': '', 'maintainer_email': '', 'home_page': 'http://www.mdsplus.org/', 'license': '', 'summary': 'MDSplus Python Objects', 'description': '\n      This module provides all of the functionality of MDSplus TDI natively in python.\n      All of the MDSplus data types such as signal are represented as python classes.\n      ', 'keywords': FieldStorage('keywords', 'physics'), 'platform': '', 'download_url': 'http://www.mdsplus.org/mdsplus_download/python', '_pypi_ordering': None, '_pypi_hidden': False, 'cheesecake_installability_id': None, 'cheesecake_documentation_id': None, 'cheesecake_code_kwalitee_id': None, 'requires_python': '', 'description_from_readme': None}]

Quick attempt at debugging this, according to the Sentry traceback the error happens when trying to insert a new Release including 'keywords': FieldStorage('keywords', 'physics') as a parameter for the query.
I believe the sequence would be:

a POST request is made
a WSGI environment is created and parsed by Webob
a Webob Request object contains a Multidict in its POST attribute from the parsed body of the request which somehow causes keywords to be a FieldStorage instance
The view constructs a MetadataForm from request.POST and successfully validates the form including the FieldStorage as a StringField.
The view then constructs and adds a new Release which causes the error.

Would it be possible to check the body of the request?