Add check for identical definitions in ontology file?

Hi @cmungall
Stemming from https://github.com/geneontology/go-ontology/issues/12814
There were 2 terms in the ontology with identical definitions (likely a results of copy-paste):
[Term]
id: GO:0043946
name: positive regulation of catalytic activity in other organism involved in symbiotic interaction
namespace: biological_process
def: "Any process in which an organism stops, prevents, or reduces the frequency, rate or extent of enzyme activity in a second organism, where the two organisms are in a symbiotic interaction." [GOC:mtg_pamgo_17jul06, GOC:tb]
[Term]
id: GO:0052199
name: negative regulation of catalytic activity in other organism involved in symbiotic interaction
namespace: biological_process
def: "Any process in which an organism stops, prevents, or reduces the frequency, rate or extent of enzyme activity in a second organism, where the two organisms are in a symbiotic interaction." [GOC:mtg_pamgo_17jul06, GOC:tb]
I fixed GO:0043946, but I’m a bit surprised that we don’t have a check in place to detect identical definitions. Because if there are any, one of them has to be incorrect (or we need to merge terms). Similar to how we have a check for identical class labels. Thanks!

@mcourtot , @ukemi
Labeling as editors-discussion as I believe it should be fairly quick to discuss and agree on having such a check - or reporting on whether it exists already but was not working. Thanks.

Someone has a thanksgiving project...

ERROR: duplicate-defs: GO:0046920 GO:0046921 --> Catalysis of the transfer of an L-fucosyl group from GDP-beta-L-fucose to an acceptor molecule to form an alpha-(1->6) linkage.
ERROR: duplicate-defs: GO:1990526 GO:1990527 --> A multiprotein complex that is involved in the transcription regulation of mating genes in the yeast S. cerevisiae.
ERROR: duplicate-defs: GO:0034413 GO:0071970 --> The chemical reactions and pathways resulting in the formation of (1->3)-beta-D-glucans, compounds composed of glucose residues linked by (1->3)-beta-D-glucosidic bonds, found in the walls of ascospores.
ERROR: duplicate-defs: GO:0003422 GO:0060536 --> The process in which the anatomical structures of cartilage are generated and organized.
ERROR: duplicate-defs: GO:0099622 GO:0099624 --> The process in which ions are transported across the plasma membrane of a cardiac muscle cell such that the membrane potential changes in the repolarizing direction, toward the steady state potential. For example, the repolarization during an action potential is from a positive membrane potential towards a negative resting potential.
ERROR: duplicate-defs: GO:0070472 GO:0070473 GO:0070474 --> Any process that modulates the frequency, rate or extent of uterine smooth muscle contraction.
ERROR: duplicate-defs: GO:0061001 GO:0061003 --> Any process that modulates the rate, frequency, or extent of dendritic spine morphogenesis, the process in which the anatomical structures of a dendritic spine are generated and organized. A dendritic spine is a protrusion from a dendrite and a specialized subcellular compartment involved in synaptic transmission.
ERROR: duplicate-defs: GO:0010767 GO:0010768 --> Any process that modulates the frequency, rate or extent of transcription from an RNA polymerase II promoter as a result of a UV damage stimulus.
ERROR: duplicate-defs: GO:0071530 GO:0071531 --> A process of protein complex assembly in which the arrangement and bonding together of the set of components that form the protein complex is mediated by an FHA (forkhead-associated) domain interaction.
ERROR: duplicate-defs: GO:0071325 GO:0071329 --> Any process that results in a change in state or activity of a cell (in terms of movement, secretion, enzyme production, gene expression, etc.) as a result of a sucrose stimulus.
ERROR: duplicate-defs: GO:0099626 GO:0099635 --> Regulation of presynaptic cytosolic calcium ion concentrations via the directed movement of calcium ions across the plasma-membrane into the cytosol via the action of a voltage-gated calcium ion channel.  This is the first step in synaptic transmission
ERROR: duplicate-defs: GO:0071989 GO:1990975 --> The directed movement of a protein to a specific location at the spindle pole body.
ERROR: duplicate-defs: GO:0001005 GO:0001157 --> The function of binding to a specific DNA sequence motif in a type 1 RNA polymerase III (Pol III) promoter in order to recruit the Pol III recruiting factor TFIIIB to the promoter. For type 1 Pol III promoters, this requires both TFIIIA and TFIIIC.
ERROR: duplicate-defs: GO:0010167 GO:0080033 --> Any process that results in a change in state or activity of a cell or an organism (in terms of movement, secretion, enzyme production, gene expression, etc.) as a result of a nitrate stimulus.
ERROR: duplicate-defs: GO:0071020 GO:0071021 GO:0071022 --> A spliceosomal complex that is formed following the second splicing event and contains the spliced product, the excised intron, and three snRNPs, including U5.
ERROR: duplicate-defs: GO:0061396 GO:0061398 --> Any process that modulates the frequency, rate or extent of transcription from an RNA polymerase II promoter in response to an copper ion stimulus.
ERROR: duplicate-defs: GO:0030148 GO:0090159 --> The chemical reactions and pathways resulting in the formation of sphingolipids, any of a class of lipids containing the long-chain amine diol sphingosine or a closely related base (a sphingoid).
ERROR: duplicate-defs: GO:0043927 GO:0070480 --> The chemical reactions and pathways resulting in the breakdown of the transcript body of a nuclear-transcribed mRNA that occurs when the ends are not protected by the 5'-cap.
ERROR: duplicate-defs: GO:0001119 GO:0001120 GO:0034367 --> The acquisition, loss, or modification of macromolecules within a complex, resulting in the alteration of an existing complex.
ERROR: duplicate-defs: GO:0010663 GO:0010664 --> Any process that increases the rate or extent of striated muscle cell apoptotic process, a form of programmed cell death induced by external or internal signals that trigger the activity of proteolytic caspases whose actions dismantle a striated muscle cell and result in its death.
ERROR: duplicate-defs: GO:0044618 GO:0044619 --> The process in which an organism effects a change in the relaxation of smooth muscle in the uterus of a second organism.
ERROR: duplicate-defs: GO:0035401 GO:0035409 --> The modification of histone H3 by the addition of a phosphate group to a tyrosine residue at position 41 of the histone.
ERROR: duplicate-defs: GO:0071583 GO:0071584 --> Any process that stops, prevents, or reduces the frequency, rate or extent of the directed movement of zinc ions (Zn2+) from one side of a membrane to the other.
ERROR: duplicate-defs: GO:0010883 GO:0010888 --> Any process that modulates the rate, frequency or extent of lipid storage. Lipid storage is the accumulation and maintenance in cells or tissues of lipids, compounds soluble in organic solvents but insoluble or sparingly soluble in aqueous solvents. Lipid reserves can be accumulated during early developmental stages for mobilization and utilization at later stages of development.
ERROR: duplicate-defs: GO:0010889 GO:0010891 --> Any process that modulates the rate, frequency or extent of sequestering of triglyceride. Triglyceride sequestration is the process of binding or confining any triester of glycerol such that it is separated from other components of a biological system.
ERROR: duplicate-defs: GO:0090216 GO:0090217 --> Any process that decreases the frequency, rate or extent of the catalysis of the reaction: ATP + 1-phosphatidyl-1D-myo-inositol 4-phosphate = ADP + 1-phosphatidyl-1D-myo-inositol 4,5-bisphosphate.
ERROR: duplicate-defs: GO:0052573 GO:0052574 --> The chemical reactions and pathways involving UDP-D-galactose, a substance composed of D-galactose in glycosidic linkage with guanosine diphosphate.
ERROR: duplicate-defs: GO:0001113 GO:0001127 --> Any process involved in the melting of the DNA hybrid of the core promoter region within the transcriptional closed complex of an RNA polymerase II preinitiation complex (PIC) to produce an open complex where the DNA duplex around the transcription initiation site is unwound to form the transcription bubble.
ERROR: duplicate-defs: GO:0001131 GO:0001140 --> Interacting selectively and non-covalently with a sequence of DNA that is in cis with and relatively close to a core promoter for a bacterial-type RNA polymerase in order to modulate transcription by bacterial-type RNA polymerase. The transcription factor may or may not also interact selectively with a protein or macromolecular complex.
ERROR: duplicate-defs: GO:0098872 GO:0099579 --> G-protein coupled neurotransmitter receptor activity occurring in the postsynaptic membrane that is part of a GPCR signaling pathway that positively regulates ion channel activity in the postsynaptic membrane.
ERROR: duplicate-defs: GO:0060331 GO:0060332 --> Any process that decreases the rate, frequency or extent of a response to interferon-gamma. Response to interferon gamma is a change in state or activity of a cell or an organism (in terms of movement, secretion, enzyme production, gene expression, etc.) as a result of an interferon-gamma stimulus.
ERROR: duplicate-defs: GO:0034675 GO:0034676 --> An integrin complex that comprises one alpha6 subunit and one beta1 subunit.
ERROR: duplicate-defs: GO:0034409 GO:0071969 --> The chemical reactions and pathways involving (1->3)-beta-D-glucans, compounds composed of glucose residues linked by (1->3)-beta-D-glucosidic bonds, found in the walls of ascospores.
ERROR: duplicate-defs: GO:0044758 GO:0044759 --> Any process in which a symbiont organism modulates the frequency, rate or extent of synaptic transmission, communication from a neuron to a target (neuron, muscle, or secretory cell) across a synapse, in its host organism.
ERROR: duplicate-defs: GO:0098881 GO:0098967 --> The exocytic fusion of neurotransmitter receptor containing vesicles with a plasma-membrane resulting in the integration of NT receptors with the plasma-membrane enabling them to participate in neurotransmitter reception.  This process includes tethering and docking steps that prepare vesicles for fusion.
ERROR: duplicate-defs: GO:0001130 GO:0001216 GO:0001217 --> Interacting selectively and non-covalently with a specific DNA sequence in order to modulate transcription by bacterial-type RNA polymerase. The transcription factor may or may not also interact selectively with a protein or macromolecular complex.
ERROR: duplicate-defs: GO:0090450 GO:1990003 --> Catalysis of the reaction: IDP + H2O = IMP + phosphate.
ERROR: duplicate-defs: GO:0000339 GO:0098808 --> Interacting selectively and non-covalently with a 7-methylguanosine (m7G) group or derivative located at the 5' end of an RNA molecule.
ERROR: duplicate-defs: GO:0044866 GO:0044869 --> The process in which an organism effects a change in symbiont exo-alpha-sialidase activity, the catalysis of the hydrolysis of peptide bonds in a protein.
ERROR: duplicate-defs: GO:0098635 GO:0098636 --> Any protein complex that is capable of carrying out some part of the process of cell-cell adhesion.
ERROR: duplicate-defs: GO:0098551 GO:0098560 --> The side (leaflet) of the late endosome membrane that faces the cytoplasm.
ERROR: duplicate-defs: GO:0003371 GO:0003372 --> Any cellular process that results in the specification, formation or maintenance of polarized cytoskeletal structures that contribute to the cell polarity of a migrating ameboid cell.
ERROR: duplicate-defs: GO:0047113 GO:0047770 --> Catalysis of the reaction: an aldehyde + acceptor + H2O = a carboxylate + reduced acceptor.
ERROR: duplicate-defs: GO:0061046 GO:0061047 --> Any process that modulates the rate, frequency, or extent of the process in which a highly ordered sequence of patterning events generates the branched structures of the lung, consisting of reiterated combinations of bud outgrowth, elongation, and dichotomous subdivision of terminal units.
ERROR: duplicate-defs: GO:0006073 GO:0044042 --> The chemical reactions and pathways involving glucans, polysaccharides consisting only of glucose residues.
ERROR: duplicate-defs: GO:0046484 GO:0052838 --> The chemical reactions and pathways involving oxazole or thiazole, five-membered heterocyclic ring structures containing an oxygen and a sulfur, respectively, in the 1-position and a nitrogen in the 3-position.
ERROR: duplicate-defs: GO:0071580 GO:0071581 --> Any process that modulates the frequency, rate or extent of the directed movement of zinc ions (Zn2+) from one side of a membrane to the other.
ERROR: duplicate-defs: GO:0008416 GO:0051750 --> Catalysis of the isomerization of 3,5-dienoyl-CoA to 2,4-dienoyl-CoA.
ERROR: duplicate-defs: GO:0001116 GO:0065004 --> The aggregation, arrangement and bonding together of proteins and DNA molecules to form a protein-DNA complex.
ERROR: duplicate-defs: GO:0003416 GO:0098867 --> The increase in size or mass of an endochondral bone that contributes to the shaping of the bone.
ERROR: duplicate-defs: GO:0080072 GO:0080089 --> Catalysis of the transfer of a sinapoyl group to a nitrogen atom on the spermidine molecule.
ERROR: duplicate-defs: GO:0042203 GO:0046252 --> The chemical reactions and pathways resulting in the breakdown of toluene, a volatile monoaromatic hydrocarbon found in crude petroleum and petroleum products.
ERROR: duplicate-defs: GO:0070908 GO:0070909 --> Catalysis of the reaction: tyrosine(out) + tyramine(in) = tyrosine(in) + tyramine(out).
ERROR: duplicate-defs: GO:0098924 GO:0098940 --> Cell-cell signaling from postsynapse to presynapse, across the synaptic cleft, mediated by nitric oxide.
ERROR: duplicate-defs: GO:0002661 GO:0010932 --> Any process that modulates the frequency, rate, or extent of B cell tolerance induction.
ERROR: duplicate-defs: GO:0009308 GO:0044106 --> The chemical reactions and pathways involving any organic compound that is weakly basic in character and contains an amino or a substituted amino group, as carried out by individual cells. Amines are called primary, secondary, or tertiary according to whether one, two, or three carbon atoms are attached to the nitrogen atom.
ERROR: duplicate-defs: GO:0070454 GO:0070455 --> Any process that activates or increases the frequency, rate or extent of the chemical reactions and pathways resulting in the formation of heme.
ERROR: duplicate-defs: GO:1902731 GO:1902732 --> Any process that stops, prevents, or reduces the frequency, rate or extent of the multiplication or reproduction of chondrocytes by cell division, resulting in the expansion of their population. A chondrocyte is a polymorphic cell that forms cartilage.
ERROR: duplicate-defs: GO:0071249 GO:0071250 --> Any process that results in a change in state or activity of a cell (in terms of movement, secretion, enzyme production, gene expression, etc.) as a result of a nitrate stimulus.
ERROR: duplicate-defs: GO:0044488 GO:0044489 --> Any process in which an organism effects a change in the frequency, rate or extent of the activity of a voltage-gated sodium channel in another organism.
ERROR: duplicate-defs: GO:1990626 GO:1990627 --> The membrane organization process that joins two mitochondrial outer membranes to form a single membrane.
ERROR: duplicate-defs: GO:0060406 GO:0060407 --> Any process that increases the rate, frequency or extent of penile erection.  Achieved by increased inflow of blood into the vessels of erectile tissue, and decreased outflow.
ERROR: duplicate-defs: GO:0009744 GO:0010555 --> Any process that results in a change in state or activity of a cell or an organism (in terms of movement, secretion, enzyme production, gene expression, etc.) as a result of a sucrose stimulus.
ERROR: duplicate-defs: GO:0070631 GO:1990944 --> Any process in which a spindle pole body is transported to, or maintained in, a specific location. A spindle pole body is a type of microtubule organizing center found in fungal cells.
ERROR: duplicate-defs: GO:0099582 GO:0099583 --> Any neurotransmitter receptor activity that is involved in regulating the concentration of calcium in the presynaptic cytosol.
ERROR: duplicate-defs: GO:0016763 GO:0052636 --> Catalysis of the transfer of a pentosyl group from one compound (donor) to another (acceptor).
ERROR: duplicate-defs: GO:0044841 GO:0044842 --> The lumen of a gut granule, a lysosome-related organelle contained within the intestinal cells of the nematode C. elegans.
ERROR: duplicate-defs: GO:1990091 GO:1990092 --> The sodium-dependent hydrolysis of proteins into smaller polypeptides and/or amino acids by cleavage of their own peptide bonds.

This is over-reporting, it doesn't attempt to clump, if a definition is shared by N classes then every permutation is reported
We should definitely have this check. I can commit this check whenever you like. I'd rather do it sooner. Obv once committed releases will be blocked until all dupes are fixed.

Put them in a spreadsheet at https://docs.google.com/spreadsheets/d/12j3dwON67zLWOhMklfeLRMWY14XAAWzdEhvNdXNpNks/edit#gid=0
If we can each do a few each time we edit we should be done quickly. Please indicate the ones that have been fixed in the last column.

I made a start. We may want to split the list among us, to avoid duplication of effort...
@mcourtot @ukemi @tberardini @dosumis

Done a few more.
@dosumis if that helps I've noted "DOS" in the last column for rows/terms that are in your ID range, thanks :-)

Ta

Ok.  Fixed everything that was left EXCEPT for the ones labeled DOS.

Added Melanie, Tanya and myself as assignees for stats purposes.

Fixed.

Brilliant, thanks @dosumis !
Handing over to @cmungall to implement the check then.
As an aside, looking at the duplicates (and at ensuing tickets) supports the view that it's safer to have ontology requests in a tracker than via TermGenie Free Form.

There are a few tickets still one - can we wait until they're closed to implement the check? Otherwise release will break.
https://github.com/geneontology/go-ontology/issues/12825
https://github.com/geneontology/go-ontology/issues/12821

I'll do as soon as #12825 is closed (the other is closed now)

https://github.com/geneontology/go-ontology/issues/12825 is closed.  @cmungall, you can proceed with the check implementation.

Check implemented. Caught its first duplicate too!  I'll close this issue.

For the record, the dupe caught was:
ERROR: duplicate-defs: GO:0099626 GO:0099635 --> Regulation of presynaptic cytosolic calcium ion concentrations via the directed movement of calcium ions across the plasma-membrane into the cytosol via the action of a voltage-gated calcium ion channel.  This is the first step in synaptic transmission


update commented link in spec_helper

Replacement for #2003, not sure what happened there.

/cc @myronmarston


Stream manager crash in stdlib/strtol_l.c

We noticed the stream manager continually crashing in 2 out of 11 containers running in aurora. This specific topology is running on heron version 0.14.4.
I don't know gdb very well but I was able to get the following information from the core dump:
#0  0x00007f302e629c37 in __GI_____strtoul_l_internal (nptr=0x7fff1cf06d20 "", endptr=0x5807b0, base=10, group=<optimized out>, loc=<optimized out>) at ../stdlib/strtol_l.c:365
365     ../stdlib/strtol_l.c: No such file or directory.
[Current thread is 1 (LWP 100082)]
(gdb) bt
#0  0x00007f302e629c37 in __GI_____strtoul_l_internal (nptr=0x7fff1cf06d20 "", endptr=0x5807b0, base=10, group=<optimized out>, loc=<optimized out>) at ../stdlib/strtol_l.c:365
#1  0x0000000000000000 in ?? ()
(gdb) where
#0  0x00007f302e629c37 in __GI_____strtoul_l_internal (nptr=0x7fff1cf06d20 "", endptr=0x5807b0, base=10, group=<optimized out>, loc=<optimized out>) at ../stdlib/strtol_l.c:365
#1  0x0000000000000000 in ?? ()
(gdb) thread apply all bt

Thread 3 (LWP 100084):
#0  0x00007f302ede7404 in __pthread_mutex_unlock_usercnt (mutex=0x1d, decr=<optimized out>) at pthread_mutex_unlock.c:55
#1  0x00007f302cb599c0 in ?? ()
#2  0x00007f302cb59700 in ?? ()
#3  0x25fc65e0e2e8d055 in ?? ()
#4  0x25fc61379db2d055 in ?? ()
#5  0x0000000000000000 in ?? ()

Thread 2 (LWP 100083):
#0  0x00007f302e6dffdd in parse_param (quoted=0, ifs_white=0x0, ifs=0xdb9c3c8bff28d055 <error: Cannot access memory at address 0xdb9c3c8bff28d055>, pwordexp=0x7f302d35a700, flags=0, offset=<optimized out>, words=<optimized out>,
    max_length=0x20000000e, word_length=<optimized out>, word=0x486656 <do_io+294>) at wordexp.c:1628
#1  parse_dollars (word=0x486656 <do_io+294>, word_length=<optimized out>, max_length=0x20000000e, words=<optimized out>, offset=<optimized out>, flags=0, pwordexp=0x7f302d35a700,
    ifs=0xdb9c3c8bff28d055 <error: Cannot access memory at address 0xdb9c3c8bff28d055>, ifs_white=0x0, quoted=0) at wordexp.c:2112
#2  0x00007f302d35a700 in ?? ()
#3  0x00007f302d35a700 in ?? ()
#4  0xdb9c3c8bff28d055 in ?? ()
#5  0x0000000000000000 in ?? ()

Thread 1 (LWP 100082):
#0  0x00007f302e629c37 in __GI_____strtoul_l_internal (nptr=0x7fff1cf06d20 "", endptr=0x5807b0, base=10, group=<optimized out>, loc=<optimized out>) at ../stdlib/strtol_l.c:365
#1  0x0000000000000000 in ?? ()
(gdb) list +
360     in ../stdlib/strtol_l.c```

This issue seemed to have persisted for a few days and the resolution was to restart the topology. If there's any other information that would be helpful or more digging I can do with gdb let me know.

As far as I know this is the first time we've seen this happen when running topologies on heron.

stdlib/strtol_l.c is from glibc, not heron,  so this is really strange. And many symbols are missing in the stack trace. What's the version of glibc on those hosts?

ldd --version
ldd (Ubuntu EGLIBC 2.19-0ubuntu6.9) 2.19
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

Does that answer your question?

I think it was missing symbols because I opened the core dump on another machine. Here is the backtrace from the core dump when opened in gdb on the same machine:
#0  0x00007f302e629c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f302e62d028 in __GI_abort () at abort.c:89
#2  0x00000000004b4717 in google::DumpStackTraceAndExit () at src/utilities.cc:150
#3  0x00000000004abaca in google::LogMessage::Fail () at src/logging.cc:1478
#4  0x00000000004ad9ff in google::LogMessage::SendToLog (this=0x7fff1cf06d20) at src/logging.cc:1432
#5  0x00000000004ab6af in google::LogMessage::Flush (this=this@entry=0x7fff1cf06d20) at src/logging.cc:1301
#6  0x00000000004ae33e in google::LogMessageFatal::~LogMessageFatal (this=0x7fff1cf06d20, __in_chrg=<optimized out>)
    at src/logging.cc:2013
#7  0x000000000040e37b in heron::stmgr::StMgr::CheckTMasterLocation(EventLoop::Status) ()
#8  0x0000000000493231 in EventLoopImpl::handleTimerCallback(long, short) ()
#9  0x0000000000490f8b in EventLoopImpl::eventLoopImplTimerCallback(int, short, void*) ()
#10 0x000000000049c39c in event_process_active_single_queue (activeq=0x1cee020, base=0x1cfe000) at event.c:1350
#11 event_process_active (base=<optimized out>) at event.c:1420
#12 event_base_loop (base=0x1cfe000, flags=0) at event.c:1621
#13 0x0000000000409416 in main ()

Yeah, it makes sense now. Stmgr fails here:
void StMgr::CheckTMasterLocation(EventLoop::Status) {
  if (!tmaster_client_) {
    LOG(FATAL) << "Could not fetch the TMaster location in time. Exiting. " << std::endl;
  }
}

@ajorgensen Can you share the log of that stmgr?

Here is the log file from one of the containers that was experiencing the issue. Its interesting that only 3 out of 10 were experiencing this problem.
dl download
Log file created at: 2017/01/02 14:56:08
Running on machine: mesos-worker2204
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0102 14:56:08.112805 121106 basics.cpp:241] Pruned log file log-files/heron-stmgr-stmgr-2.mesos-worker2204.answers.log.INFO.20170102-144307.118825
I0102 14:56:08.113024 121106 stmgr.cpp:80] Init Stmgr
I0102 14:56:08.218334 121106 heron-zkstatemgr.cpp:546] Setting watch on tmaster location
I0102 14:56:08.218350 121106 zkclient.cpp:177] Checking if /heron/tmasters/topology exists
I0102 14:56:08.218374 121106 stmgr.cpp:175] Fetching TMaster Location
I0102 14:56:08.218380 121106 zkclient.cpp:239] Getting zknode /heron/tmasters/topology
I0102 14:56:08.218400 121106 stmgr.cpp:187] Creating StmgrServer
I0102 14:56:08.218592 121106 stmgr.cpp:217] Creating tuple cache
I0102 14:56:08.218633 121106 metricsmgr-client.cpp:49] Connected to metrics manager
I0102 14:56:08.219760 121106 metricsmgr-client.cpp:86] Successfully registered ourselves to the metricsmgr
I0102 14:56:08.219766 121106 metricsmgr-client.cpp:95] Do not have a TMasterLocation yet
I0102 14:56:08.221850 121108 zkclient.cpp:307] ZKClient GlobalWatcher called with type SESSION_EVENT and state CONNECTED_STATE
I0102 14:56:08.221858 121108 zkclient.cpp:318] Got a new session id:<PHONE_NUMBER>25614812
I0102 14:56:08.222769 121106 heron-zkstatemgr.cpp:523] Setting watch on tmaster location succeeded: no node
E0102 14:56:08.222812 121106 heron-zkstatemgr.cpp:304] Error getting tmaster location because the tmaster does not exist
I0102 14:56:08.222817 121106 stmgr.cpp:261] TMaster Location Fetch failed with status 2000
I0102 14:56:08.222820 121106 stmgr.cpp:262] Retrying after 10000000 micro seconds
I0102 14:56:13.096352 121106 stmgr-server.cpp:171] Got new connection 0x187fe00 from <IP_ADDRESS>:45406
I0102 14:56:13.096390 121106 stmgr-server.cpp:171] Got new connection 0x187ff80 from <IP_ADDRESS>:45407
I0102 14:56:13.096618 121106 stmgr-server.cpp:267] Got HandleRegisterInstanceRequest from connection 0x187ff80 and instance container_2_topology-events_12
I0102 14:56:13.096623 121106 stmgr-server.cpp:306] New instance registered with us container_2_topology-events_12
I0102 14:56:13.096686 121106 stmgr-server.cpp:267] Got HandleRegisterInstanceRequest from connection 0x187fe00 and instance container_2_bolt_32
I0102 14:56:13.096691 121106 stmgr-server.cpp:306] New instance registered with us container_2_bolt_32
I0102 14:56:13.096849 121106 stmgr-server.cpp:171] Got new connection 0x1880100 from <IP_ADDRESS>:45409
I0102 14:56:13.097143 121106 stmgr-server.cpp:267] Got HandleRegisterInstanceRequest from connection 0x1880100 and instance container_2_topology-ios-events_2
I0102 14:56:13.097149 121106 stmgr-server.cpp:306] New instance registered with us container_2_topology-ios-events_2
I0102 14:56:13.098670 121106 stmgr-server.cpp:171] Got new connection 0x1880280 from <IP_ADDRESS>:45410
I0102 14:56:13.098938 121106 stmgr-server.cpp:267] Got HandleRegisterInstanceRequest from connection 0x1880280 and instance container_2_topology-events_22
I0102 14:56:13.098945 121106 stmgr-server.cpp:306] New instance registered with us container_2_topology-events_22
I0102 14:56:13.098958 121106 stmgr.cpp:315] We haven't received tmaster location yet, so tmaster_client_ hasn't been createdOnce we get the location, it will be started
I0102 14:56:18.223721 121106 stmgr.cpp:175] Fetching TMaster Location
I0102 14:56:18.223739 121106 zkclient.cpp:239] Getting zknode /heron/tmasters/topology
E0102 14:56:18.224812 121106 heron-zkstatemgr.cpp:304] Error getting tmaster location because the tmaster does not exist
I0102 14:56:18.224817 121106 stmgr.cpp:261] TMaster Location Fetch failed with status 2000
I0102 14:56:18.224819 121106 stmgr.cpp:262] Retrying after 10000000 micro seconds
I0102 14:56:28.225303 121106 stmgr.cpp:175] Fetching TMaster Location
I0102 14:56:28.225322 121106 zkclient.cpp:239] Getting zknode /heron/tmasters/topology
E0102 14:56:28.226440 121106 heron-zkstatemgr.cpp:304] Error getting tmaster location because the tmaster does not exist
I0102 14:56:28.226447 121106 stmgr.cpp:261] TMaster Location Fetch failed with status 2000
I0102 14:56:28.226449 121106 stmgr.cpp:262] Retrying after 10000000 micro seconds
I0102 14:56:38.226729 121106 stmgr.cpp:175] Fetching TMaster Location
I0102 14:56:38.226747 121106 zkclient.cpp:239] Getting zknode /heron/tmasters/topology
E0102 14:56:38.227967 121106 heron-zkstatemgr.cpp:304] Error getting tmaster location because the tmaster does not exist
I0102 14:56:38.227973 121106 stmgr.cpp:261] TMaster Location Fetch failed with status 2000
I0102 14:56:38.227975 121106 stmgr.cpp:262] Retrying after 10000000 micro seconds
I0102 14:56:48.228669 121106 stmgr.cpp:175] Fetching TMaster Location
I0102 14:56:48.228688 121106 zkclient.cpp:239] Getting zknode /heron/tmasters/topology
E0102 14:56:48.229796 121106 heron-zkstatemgr.cpp:304] Error getting tmaster location because the tmaster does not exist
I0102 14:56:48.229804 121106 stmgr.cpp:261] TMaster Location Fetch failed with status 2000
I0102 14:56:48.229805 121106 stmgr.cpp:262] Retrying after 10000000 micro seconds
I0102 14:56:58.230870 121106 stmgr.cpp:175] Fetching TMaster Location
I0102 14:56:58.230887 121106 zkclient.cpp:239] Getting zknode /heron/tmasters/topology
E0102 14:56:58.231976 121106 heron-zkstatemgr.cpp:304] Error getting tmaster location because the tmaster does not exist
I0102 14:56:58.231984 121106 stmgr.cpp:261] TMaster Location Fetch failed with status 2000
I0102 14:56:58.231988 121106 stmgr.cpp:262] Retrying after 10000000 micro seconds
I0102 14:57:08.232578 121106 stmgr.cpp:175] Fetching TMaster Location
I0102 14:57:08.232589 121106 zkclient.cpp:239] Getting zknode /heron/tmasters/topology
E0102 14:57:08.233913 121106 heron-zkstatemgr.cpp:304] Error getting tmaster location because the tmaster does not exist
I0102 14:57:08.233921 121106 stmgr.cpp:261] TMaster Location Fetch failed with status 2000
I0102 14:57:08.233922 121106 stmgr.cpp:262] Retrying after 10000000 micro seconds
I0102 14:57:18.234390 121106 stmgr.cpp:175] Fetching TMaster Location
I0102 14:57:18.234408 121106 zkclient.cpp:239] Getting zknode /heron/tmasters/topology
E0102 14:57:18.235612 121106 heron-zkstatemgr.cpp:304] Error getting tmaster location because the tmaster does not exist
I0102 14:57:18.235620 121106 stmgr.cpp:261] TMaster Location Fetch failed with status 2000
I0102 14:57:18.235622 121106 stmgr.cpp:262] Retrying after 10000000 micro seconds
I0102 14:57:28.235913 121106 stmgr.cpp:175] Fetching TMaster Location
I0102 14:57:28.235932 121106 zkclient.cpp:239] Getting zknode /heron/tmasters/topology
E0102 14:57:28.237000 121106 heron-zkstatemgr.cpp:304] Error getting tmaster location because the tmaster does not exist
I0102 14:57:28.237009 121106 stmgr.cpp:261] TMaster Location Fetch failed with status 2000
I0102 14:57:28.237010 121106 stmgr.cpp:262] Retrying after 10000000 micro seconds
I0102 14:57:38.237598 121106 stmgr.cpp:175] Fetching TMaster Location
I0102 14:57:38.237618 121106 zkclient.cpp:239] Getting zknode /heron/tmasters/topology
E0102 14:57:38.238720 121106 heron-zkstatemgr.cpp:304] Error getting tmaster location because the tmaster does not exist
I0102 14:57:38.238726 121106 stmgr.cpp:261] TMaster Location Fetch failed with status 2000
I0102 14:57:38.238729 121106 stmgr.cpp:262] Retrying after 10000000 micro seconds
I0102 14:57:48.239404 121106 stmgr.cpp:175] Fetching TMaster Location
I0102 14:57:48.239424 121106 zkclient.cpp:239] Getting zknode /heron/tmasters/topology
E0102 14:57:48.240520 121106 heron-zkstatemgr.cpp:304] Error getting tmaster location because the tmaster does not exist
I0102 14:57:48.240530 121106 stmgr.cpp:261] TMaster Location Fetch failed with status 2000
I0102 14:57:48.240533 121106 stmgr.cpp:262] Retrying after 10000000 micro seconds
I0102 14:57:58.240674 121106 stmgr.cpp:175] Fetching TMaster Location
I0102 14:57:58.240692 121106 zkclient.cpp:239] Getting zknode /heron/tmasters/topology
E0102 14:57:58.241816 121106 heron-zkstatemgr.cpp:304] Error getting tmaster location because the tmaster does not exist
I0102 14:57:58.241823 121106 stmgr.cpp:261] TMaster Location Fetch failed with status 2000
I0102 14:57:58.241827 121106 stmgr.cpp:262] Retrying after 10000000 micro seconds
F0102 14:58:08.219342 121106 stmgr.cpp:152] Could not fetch the TMaster location in time. Exiting. ```

Here is a snippet of a log from another container:

I0102 15:12:43.855743 114846 zkclient.cpp:239] Getting zknode /heron/tmasters/topology
E0102 15:12:43.856377 114846 heron-zkstatemgr.cpp:304] Error getting tmaster location because the tmaster does not exist
I0102 15:12:43.856384 114846 stmgr.cpp:261] TMaster Location Fetch failed with status 2000
I0102 15:12:43.856386 114846 stmgr.cpp:262] Retrying after 10000000 micro seconds
W0102 15:12:43.873224 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-3 running at mesos-worker2084:31423 due to: 1
I0102 15:12:43.873244 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:44.226778 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-5 running at mesos-worker2030:31673 due to: 1
I0102 15:12:44.226799 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:44.227692 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-2 running at mesos-worker2030:31427 due to: 1
I0102 15:12:44.227700 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:44.876132 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-3 running at mesos-worker2084:31423 due to: 1
I0102 15:12:44.876155 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:45.227958 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-5 running at mesos-worker2030:31673 due to: 1
I0102 15:12:45.227978 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:45.228839 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-2 running at mesos-worker2030:31427 due to: 1
I0102 15:12:45.228845 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:45.877647 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-3 running at mesos-worker2084:31423 due to: 1
I0102 15:12:45.877677 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:46.229532 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-5 running at mesos-worker2030:31673 due to: 1
I0102 15:12:46.229552 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:46.230432 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-2 running at mesos-worker2030:31427 due to: 1
I0102 15:12:46.230439 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:46.879498 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-3 running at mesos-worker2084:31423 due to: 1
I0102 15:12:46.879516 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:47.231680 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-5 running at mesos-worker2030:31673 due to: 1
I0102 15:12:47.231698 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:47.233083 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-2 running at mesos-worker2030:31427 due to: 1
I0102 15:12:47.233088 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:47.881393 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-3 running at mesos-worker2084:31423 due to: 1
I0102 15:12:47.881440 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:48.233335 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-5 running at mesos-worker2030:31673 due to: 1
I0102 15:12:48.233353 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:48.234001 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-2 running at mesos-worker2030:31427 due to: 1
I0102 15:12:48.234007 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:48.882557 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-3 running at mesos-worker2084:31423 due to: 1
I0102 15:12:48.882612 114846 stmgr-client.cpp:107] Retrying again...
I0102 15:12:49.215821 114846 tmaster-client.cpp:186] Sending heartbeat
W0102 15:12:49.235723 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-5 running at mesos-worker2030:31673 due to: 1
I0102 15:12:49.235741 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:49.235965 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-2 running at mesos-worker2030:31427 due to: 1
I0102 15:12:49.235970 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:49.884268 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-3 running at mesos-worker2084:31423 due to: 1
I0102 15:12:49.884287 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:50.238018 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-5 running at mesos-worker2030:31673 due to: 1
I0102 15:12:50.238035 114846 stmgr-client.cpp:107] Retrying again...
W0102 15:12:50.238272 114846 stmgr-client.cpp:99] Could not connect to stmgr stmgr-2 running at mesos-worker2030:31427 due to: 1

Here is a snippet from the tmaster logs:

I1219 23:26:33.567201 95596 physical-plan-helper.cpp:85] Topology State: 1
I1219 23:26:33.567203 95596 stmgrstate.cpp:93] Sending a new physical plan to stmgr stmgr-1
I1219 23:26:33.567247 95596 stmgrstate.cpp:93] Sending a new physical plan to stmgr stmgr-10
I1219 23:26:33.567276 95596 stmgrstate.cpp:93] Sending a new physical plan to stmgr stmgr-2
I1219 23:26:33.567303 95596 stmgrstate.cpp:93] Sending a new physical plan to stmgr stmgr-3
I1219 23:26:33.567332 95596 stmgrstate.cpp:93] Sending a new physical plan to stmgr stmgr-4
I1219 23:26:33.567358 95596 stmgrstate.cpp:93] Sending a new physical plan to stmgr stmgr-5
I1219 23:26:33.567384 95596 stmgrstate.cpp:93] Sending a new physical plan to stmgr stmgr-6
I1219 23:26:33.567411 95596 stmgrstate.cpp:93] Sending a new physical plan to stmgr stmgr-7
I1219 23:26:33.567438 95596 stmgrstate.cpp:93] Sending a new physical plan to stmgr stmgr-8
I1219 23:26:33.567463 95596 stmgrstate.cpp:93] Sending a new physical plan to stmgr stmgr-9
E1225 21:56:53.524971 95596 packet.cpp:152] Remote end has done a shutdown
I1225 21:56:53.526180 95596 tmaster.cpp:573] StMgr stmgr-4 disconnected from us
E1225 21:56:53.531491 95596 packet.cpp:152] Remote end has done a shutdown
W1225 21:56:53.531535 95596 tmasterserver.cpp:54] Unknown connection closed on us from <IP_ADDRESS>:57467, possibly metrics mgr
I1225 21:57:04.400070 95596 tmaster.cpp:359] Got a register stmgr request from stmgr-4
I1225 21:57:04.400094 95596 tmaster.cpp:401] All stmgrs have connected with us
I1225 21:57:04.400393 95596 zkclient.cpp:266] Setting zknode /heron/pplans/topology
I1225 21:57:04.406215 95596 tmaster.cpp:459] Successfully wrote new assignment to state
I1225 21:57:04.406246 95596 tmaster.cpp:480] To distribute new pplan:
I1225 21:57:04.406250 95596 physical-plan-helper.cpp:62] Printing Physical Plan
I1225 21:57:04.406252 95596 physical-plan-helper.cpp:63] Topology Name: topology
I1225 21:57:04.406255 95596 physical-plan-helper.cpp:64] Topology Id: topology22da2412-7d2a-456d-ace6-54beb8854254

It looks like maybe the tmaster zknode expired and was not re-initialized? I am not sure why some nodes handled that better and other did not. But it looks like both the containers that had the issue and the ones that did not both detected that the tmaster zknode no longer existed.
Its possible that the containers that were having issues crashed for some other reason and then could not re-start because the tmaster node did not exist. I do not see anything in the tmaster logs that indicates there was a problem talking to zookeeper.
Do you know how that /heron/tmasters/ node is managed? Does it automatically go away when the connection between the tmaster and zookeeper is terminated? Is it possible that the connection was temporarily dropped due to some networking interference and the tmaster just never reconnected?

@ajorgensen - wondering if there is a network connectivity/partitioning issue here?

Its definitely possible and should be expected given we are operating in AWS although a prolonged outage would be unexpected. Does the tmaster gracefully re-connect if the connection is terminated momentarily for some reason? I would be surprised to see separate machine segmented form the entire zookeeper ensemble for multiple days.
My hypothesis is that there was a disruption between the tmaster and the zookeeper cluster that resulted in the znode being removed and never re-established but I dont know exactly how to prove that other than manually causing a network segmentation against a topology either in production or running locally.

@ajorgensen - valid point. We need to check if the tmaster is reconnecting if the partitioning happens or not?

The tmaster will try to re-connect automatically; too many failures will lead to the failure of heron-executor, which will then be re-scheduled to another host. So does the stream mgr.

If it reconnects does it restore the znode for the tmaster?
On Wed, Jan 4, 2017 at 7:34 PM Maosong Fu<EMAIL_ADDRESS>wrote:

The tmaster will try to re-connect automatically; too many failures will
lead to the failure of heron-executor, which will then be re-scheduled to
another host. So does the stream mgr.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/twitter/heron/issues/1653#issuecomment-270528424, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAfHU4u4MzAVN-1Eo55YO3dYWYij3MZfks5rPDp3gaJpZM4LZJLj
.

The znode of TMasterLocation is an ephemeral node so it will be cleaned automatically when the connection between TMaster and zookeeper breaks. And when TMaster restarts/reconnects, it will try to write its location to the same node.

Cool that sounds good. I can try to launch the topology locally and segment
the connection between zookeeper and the tmaster and nodes to see if I can
reproduce what we saw. Might just be an edge case somewhere because as far
as I know this is the only time I've seen it happen.
On Wed, Jan 4, 2017 at 8:41 PM Maosong Fu<EMAIL_ADDRESS>wrote:

The znode of TMasterLocation is an ephemeral node so it will be cleaned
automatically when the connection between TMaster and zookeeper breaks. And
when TMaster restarts/reconnects, it will try to write its location to the
same node.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/twitter/heron/issues/1653#issuecomment-270541008, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAfHU0c3DpUFx4i2pOC-ACgZvj4QW_3Iks5rPEpfgaJpZM4LZJLj
.

We've had this happen a few more times over the last 5 days so I am going to look into this a little more. I was able to segment the tmaster from zookeeper locally and confirm that the znode was properly recreated. I'm going to keep looking to see if i can find the case where this error occurs.

I was able to find a running topology that showed this issue. I was able to confirm that the tmaster is running but there is no zknode in /heron/tmasters/<topology-name>
I1116 20:06:44.640955 127856 stmgrstate.cpp:93] Sending a new physical plan to stmgr stmgr-1
I1116 20:06:44.641252 127856 stmgrstate.cpp:93] Sending a new physical plan to stmgr stmgr-2
I1116 20:06:44.684420 127856 metricsmgr-client.cpp:86] Successfully registered ourselves to the metricsmgr
I1116 20:06:44.684438 127856 metricsmgr-client.cpp:92] Sending TMaster Location to metricsmgr
I1130 21:11:31.130848 127880 zkclient.cpp:307] ZKClient GlobalWatcher called with type SESSION_EVENT and state CONNECTING_STATE
I1130 21:11:31.130864 127880 zkclient.cpp:337] Re-connecting to the zookeeper
I1130 21:11:31.134097 127880 zkclient.cpp:307] ZKClient GlobalWatcher called with type SESSION_EVENT and state CONNECTED_STATE

So it looks like there is some edge case here where the tmaster does not properly re-create its znode. Im not sure if its in there interaction between zookeeper and the tmaster or whether there is actually a branch that doesn't re-create the node after re-initializing the connection.

I think I am able to reliably reproduce this locally.

Deploy a topology locally with zookeeper as the state manager
Stop zookeeper running for 30 seconds
Restart zookeeper

You should see the tmaster reconnect to zookeeper and then the stream manager disconnect from the tmaster. The tmaster never re-establishes its znode in zookeeper and so the stream manager cannot properly reconnect.

I also found something interesting here which might be causing the issue:

try {
zk.create(path, data, CreateMode.EPHEMERAL)
} catch {
// The ephemeral node hasn't been deleted yet.
e: NodeExistsException => zk.set(path, data)
}

Something weird was happening with this code. I would restart the service, and when the the second case was triggered (the ephemeral node hadn't been deleted yet), after a little while the ZooKeeper node with my data would disappear, as if my session had expired!
Of course the reason the node disappeared is obvious in hindsight: if you didn't create the ephemeral node in your session, your session doesn't own the node, so when the owning session expires, the node will be deleted. This situation becomes more common when you don't close your ZooKeeper session explicitly when you shut down your service, so you're relying on the ZooKeeper server quorum itself to expire ephemeral nodes. And if your server can restart faster than the session timeout, usually 30 seconds, then you'll run into this situation.

Thanks @ajorgensen - for the detailed investigation. This is indeed very helpful. @congwang @ajorgensen - can you work together to identify the case and find a fix?

Nice analysis! If I understand correctly, the following should fix it?
diff --git a/heron/common/src/cpp/zookeeper/zkclient.cpp b/heron/common/src/cpp/zookeeper/zkclient.cpp
index a833f2e..0481273 100644
--- a/heron/common/src/cpp/zookeeper/zkclient.cpp
+++ b/heron/common/src/cpp/zookeeper/zkclient.cpp
@@ -198,6 +198,8 @@ void ZKClient::CreateNode(const std::string& _node, const std::string& _value, b
                           VCallback<sp_int32> cb) {
   sp_int32 flags = 0;
   if (_is_ephimeral) {
+    zoo_adelete(zk_handle_, _node.c_str(), -1, VoidCompletionWatcher,
+                CreateCallback(this, &ZKClient::ZkActionCb, std::move(cb)));
     flags |= ZOO_EPHEMERAL;
   }
   LOG(INFO) << "Creating zknode " << _node << std::endl;

I am not sure. If the problem is that the tmaster connection resets and it immediately tries to write the znode before the old one has expired then yes I think that this diff will help. I can apply this locally and give it a try.

👍

On Jan 10, 2017, at 11:58 AM, Andrew Jorgensen<EMAIL_ADDRESS>wrote:
I am not sure. If the problem is that the tmaster connection resets and it immediately tries to write the znode before the old one has expired then yes I think that this diff will help. I can apply this locally and give it a try.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

I tried that diff, the issue is that the node just gets created over and over again because the CallWatcher gets called with a DELETED_EVENT which looks to put the tmaster into an endless loop recreating the zknode. I'll see if i can suss out how to work around this.

I tried the change locally but looking at the code it looks like it should actually be doing the right thing so i'll need to look at it a bit more.
As another data point i've also seen this happen:
E0112 15:14:40.304669 65510 heron-zkstatemgr.cpp:359] Error getting topology because the topology does not exist
E0112 15:14:40.304677 65510 tmaster.cpp:224] For topology crash-session-tracing Error getting topology Errsettingorcode is 2000```

I've seen this happen a few times where the topologies node is removed from zookeeper somehow which causes the issues with the tmaster starting up and it can go into an endless loop.

I haven't really been able to reproduce this locally, however one extra piece of information from a recent incident is that the zookeeper server seems to have noticed that the connection to zookeeper timed out but the tmaster did not:
I0112 17:21:39.505532 50349 stmgrstate.cpp:93] Sending a new physical plan to stmgr stmgr-1
I0112 17:22:59.248440 50365 zkclient.cpp:307] ZKClient GlobalWatcher called with type SESSION_EVENT and state CONNECTING_STATE
I0112 17:22:59.248457 50365 zkclient.cpp:337] Re-connecting to the zookeeper
I0112 17:23:02.334136 50365 zkclient.cpp:307] ZKClient GlobalWatcher called with type SESSION_EVENT and state CONNECTED_STATE
E0112 22:55:32.104410 50349 packet.cpp:152] Remote end has done a shutdown
I0112 22:55:32.104485 50349 tmaster.cpp:573] StMgr stmgr-1 disconnected from us
E0113 02:27:28.202004 50349 packet.cpp:152] Remote end has done a shutdown
W0113 02:27:28.202072 50349 tmasterserver.cpp:54] Unknown connection closed on us from <IP_ADDRESS>:34466, possibly metrics mgr

and from zookeepers perspective:
2017-01-12 22:55:32,001 [myid:4] - INFO  [ProcessThread(sid:4 cport:-1)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x1586177286bfece
2017-01-12 22:55:32,003 [myid:4] - INFO  [CommitProcessor:4:NIOServerCnxn@1008] - Closed socket connection for client /<IP_ADDRESS>:34644 which had sessionid 0x1586177286bfece

I am not sure what happened between 17:23:02 and 22:55:32 on the tmaster but it seems like it was no longer able to communicate with zookeeper or the stream manager and then never recovered.


imgData is not defined

Something somewhere changed and now the demo doesn't render.

Nevermind -- I just forgot how to use it myself.


Add error handling to all view models

Without proper error handling in the view models HTTP errors can terminat subscriptions of the UI which leads wo weird user experience

should be fixed automatically when all screen are refactored to SwiftUI

all screens are now refactored to SwiftUI -> Done!


Translation of "Select a level" doesn't work

Change language from English to Russian.
Expected: text "Select a level" changed to "Выбери уровень".
Current: text remains English "Select a level".

Translation strings for "Select a level" is present in file strings.js.
  'select-a-level': {
    '__desc__': 'The prompt to select a level on the drop down view',
    'en_US': 'Select a level',
    'de_DE': 'Level auswählen',
    'zh_CN': '选择一关',
    'zh_TW': '選擇其中一關',
    'es_AR': 'Seleccioná un nivel',
    'pt_BR': 'Selecione um nível',
    'fr_FR': 'Choisissez un niveau',
    'ru_RU': 'Выбери уровень',
    'uk': 'Обери рівень'
  },

Ah yeah, this is because we cache the instance of the modal terminal and even though we re-render for locale store / level store updates, it wont reset the title of the terminal.
lemme see if I can fix this!

fixed! wish these were react components :(

Thanks for the report -- this was broken forever


Issue: URL Rewrite using web.config is broken in RC2

I have tried moving the web.config to and fro and cannot get the URL rewrite to work in IIS Express from Visual Studio 2015. If there was middleware to perform what is spelled out in the web.config snippet I could punt on the web.config and selfhost. Maybe someone can point me in that direction to solve our problem.
We need to use Angular 1.x HTML 5 routing in our application and this is how we had solved it in RC1 but it doesn't work in RC2. Others are having trouble as well: http://stackoverflow.com/questions/37304113/url-rewriting-in-asp-net-core-rc2#
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified"/>
    </handlers>
    <aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" forwardWindowsAuthToken="false"/>
    <rewrite>
      <rules>
        <rule name="AngularJS Routes" stopProcessing="true">
          <match url=".*" />
          <conditions logicalGrouping="MatchAll">
            <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
            <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />
            <add input="{REQUEST_URI}" pattern="api/(.*)" negate="true" />
          </conditions>
          <action type="Rewrite" url="/" />
        </rule>
      </rules>
    </rewrite>
  </system.webServer>
</configuration>

I'm going to ping @Ciel on Slack, who may have a solution for this.

I'll take a look at this when I get home this evening, sure. I'm using
angular extensively with RC2. I know I've got a working demo of this in one
of my projects somewhere.
On May 25, 2016 11:13 AM, "Luke Latham"<EMAIL_ADDRESS>wrote:

I'm going to ping @Ciel https://github.com/Ciel on Slack, who may have
a solution for this.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/aspnet/IISIntegration/issues/192#issuecomment-221623516

I'm not at a computer right now, but looking at your sample have you tried using "^(.*)$" as the match url? And of course, you can confirm that you've got html5mode active?

I haven't tried the regex  "^(.*)$". Any links to your examples would be great!

@anwalkers Were you able to verify the regex suggested by @ciel?

@muratg I have not. I am on a deadline and have turned HTML5 routing off temporarily. Will tackle this when I get to the Azure WebApp deployment step. Hoping @ciel will have a link to an example along the way.
That said, any progress on getting IIS Express from VS 2015 to pay attention to URL rewriting in the web.config?

I tried the following code in my Startup.cs to do middleware URL rewrite. It works fine for the base URL, but does not support deep links into an AngularJS 1.x application with HTML5 routing enabled. It does work with HTML5 routing disabled.
Works:
Base URL: http://localhost:5000
API: http://localhost:5000/api/someApi
Library or resource: http://localhost:5000/lib/library.js
HTML5 Routing Disabled: http://localhost:5000/#/angularRoute NOTE: Deep linking works here.
Does not work:
HTML5 Routing Enabled: http://localhost:5000/angularRoute NOTE: This does work if you have loaded the application from the base URL and navigated to the route, but does not work if you have pasted the deep link with the angular route, http://localhost:5000/angularRoute, into a new browser window.
From Startup.cs:
            app.UseMvc();

            // this serves "index.html" from the wwwroot folder when
            // a route not containing a file extension is not handled by MVC.
            app.MapWhen(context =>
            {
                var path = context.Request.Path.Value.ToLower();
                return path.Contains(".*");
            },
                branch =>
                {
                    branch.Use((context, next) =>
                    {
                        context.Request.Path = new PathString("/index.html");
                        return next();
                    });

                    branch.UseStaticFiles();
                });

Just for clarity, here is a snippet for enabling HTML5 routing in angular 1.x application. It removes the # hash based routing.
            $locationProvider
                .html5Mode(true);

@Tratcher thoughts on why this may have regressed?
cc @pan-wang

I have published a simple sample app to this repository.
https://github.com/anwalkers/AngularUrlRewriteAspNetCore1
This site is a vanilla ASP.Net Core 1 RC2 project using one MVC route to an index.cshtml as the root page for the site. It does not have the mapWhen function or static html in the wwwroot directory. It is checked in a broken state. You can switch it back to non-HTML5 mode by editing the site.js and a bit of markup.
It also does not have mapWhen in the Startup.cs, wasn't sure how to mix the MVC routing and the mapWhen...

@muratg @Tratcher @pan-wang @ciel  Any progress or discussion on this issue?

I was able to repro this using just the UrlRewrite samples here:
http://www.iis.net/learn/extensions/url-rewrite-module/creating-rewrite-rules-for-the-url-rewrite-module
(against ANCM rather than Asp.Net 4).
I see two issues:

Kestrel receives the request, but the url has not been modified.
Kestrel's response is not forwarded to the client, the client only receives an empty 200 response (Content-Length: 0)

Assigning to @pan-wang to debug.

Has anyone found a work around?

Unfortunately I haven't
On Jun 6, 2016 6:09 PM, "Michael Whittenburg"<EMAIL_ADDRESS>wrote:

Has anyone found a work around?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aspnet/IISIntegration/issues/192#issuecomment-224116050,
or mute the thread
https://github.com/notifications/unsubscribe/AAavtJ85Mf7Z36px_PWpZ0USzHGFCazyks5qJKiRgaJpZM4ImD44
.

I did find one where you can change type="Rewrite" to type="Redirect", but I doubt that's relevant to you here.

Looking at it now

FYI... Found this project with some middleware to solve the url rewrite problem.  Something like this would make a nice addition to ASP.Net Core 1.
https://github.com/BrainCrumbz/AWATTS/tree/master/src/WebPackAngular2TypeScript/Routing

this is a bug in aspnetcore module. The fix will be included for next release.

@pan-wang can you provide a brief explanation please?

@anwalkers I'm trying to see if I can get that middleware implemented into my project as a workaround but appears to be for RC1. Did you try it with an RC2 project?
@pan-wang Don't suppose you know when this release is expected to drop?

I was able to use this as a quick workaround.
routes.MapRoute("spa-routes", "{*anything}", new { controller = "Home", action = "Index" });

@WhittenMike807 I think this is working for me (have run into the next hurdle while upgrading to RC2).
Are there any known pitfalls of taking this approach?

I haven't encountered any issue so far. Seems like a "ok" fix for now

FYI, this also blows up and prevents debugging from starting. (works fine in production)
<rewrite>
  <rules>
    <rule name="Redirect HTTP to HTTPS" stopProcessing="true">
      <match url="^(.*)$"/>
      <conditions>
        <add input="{HTTPS}" pattern="^OFF$"/>
        <add input="{HTTP_HOST}" matchType="Pattern" pattern="^localhost(:\d+)?$" negate="true" />
        <add input="{HTTP_HOST}" matchType="Pattern" pattern="^127\.0\.0\.1(:\d+)?$" negate="true" />
      </conditions>
      <action type="Redirect" url="https://{HTTP_HOST}/{R:1}" redirectType="SeeOther"/>
    </rule>
  </rules>
</rewrite>

@Tratcher Why is this closed? Has it been updated/fixed?

Yes, this has been fixed for the upcoming release.

@Tratcher Just so I can plan. You have a rough idea on when the next release will be by chance?

https://github.com/aspnet/Home/wiki/Roadmap

Can you elaborate on fixed? Will it continue to work exactly the way it has previously with RC1? Getting used to 'fixed with side effects' and just want to be sure.

My URL rewriting is still broken with the RTM release. Wasn't this intended to be fixed?

@twilliamsgsnetx yes this was fixed in the 1.0 version of AspNetCoreModule. Have you've installed the new tooling?

@Tratcher I have uninstalled all my previous .NET Core stuff and installed all the new versions, including visual studio tooling.
Could IIS Express be utilizing an older version still somehow? How do I validate my version?

Look in (MySolution).vs\config\applicationhost.config and find <add name="AspNetCoreModule" image="%IIS_BIN%\aspnetcore.dll" />. If it says %IIS_BIN%\aspnetcore.dll you should be fine. This maps to C:\Program Files (x86)\IIS Express\aspnetcore.dll who's file version should be 7.1.1967.

Yes. Mine is set that way, and my file is the version you specified.
For clarification, my project's web.config is as follows
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
    </handlers>
    <aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" forwardWindowsAuthToken="true" stdoutLogEnabled="false" />
  </system.webServer>
</configuration>

And my wwwroot config that has the url rewriting logic is as follows:

@ZestyBread can you try this?

The web.config file doesn't exist in wwwroot anymore. Try combining the two together like:
<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
    </handlers>
    <aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" forwardWindowsAuthToken="true" stdoutLogEnabled="false" />
    <rewrite>
      <rules>
        <!--Redirect selected traffic to index -->
        <rule name="Index Rule" stopProcessing="true">
          <match url=".*" />
          <conditions logicalGrouping="MatchAll">
            <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
            <add input="{REQUEST_URI}" matchType="Pattern" pattern="^/api/" negate="true" />
            <add input="{REQUEST_URI}" matchType="Pattern" pattern="^/account/" negate="true" />
          </conditions>
          <action type="Rewrite" url="/index.html" />
        </rule>
      </rules>
    </rewrite>
  </system.webServer>
</configuration>

@ZestyBread Consolidating to my web.config in the main project as you suggested, my entire app breaks. The client files aren't served properly and angular simply doesn't load.
The rewrite section is specifically what seems to be causing it. I comment that out and everything is fine, except the inability to refresh the site on an angular router URL properly.
I hope I'm not the only one experiencing this issue.
Resource interpreted as Stylesheet but transferred with MIME type text/html:"http://localhost:1412/app/content/css/normalize.css". localhost/:8
Resource interpreted as Stylesheet but transferred with MIME type text/html:"http://localhost:1412/app/content/css/main.css". localhost/:10
Resource interpreted as Stylesheet but transferred with MIME type text/html:"http://localhost:1412/lib/css/vendors.css". shim.min.js:1
Uncaught SyntaxError: Unexpected token < zone.js:1
Uncaught SyntaxError: Unexpected token < Reflect.js:1
Uncaught SyntaxError: Unexpected token < system.src.js:1
Uncaught SyntaxError: Unexpected token < systemjs.config.js:1
Uncaught SyntaxError: Unexpected token < vendors.js:1
Uncaught SyntaxError: Unexpected token < (index):23
Uncaught ReferenceError: System is not defined

Ah, I know what's wrong here. We moved web.config and the app base dir, so rules with file constraints like matchType="IsFile" are looking in the wrong directory.
Workaround: for IIS you can move the web.config back to the wwwroot dir. https://github.com/aspnet/IISIntegration/issues/164
Real solution: @ZestyBread is writing a url-rewrite middleware so we can move this directly into Asp.Net.

Closing this one. https://github.com/aspnet/BasicMiddleware/issues/43 is tracking the middleware work.


cc/UseDartApi test is flaky: crash on simbc64 in release mode.

Log output:
Expected: Pass
Actual: Crash
CommandOutput[run_vm_unittest]:

Command[run_vm_unittest]: DART_CONFIGURATION=ReleaseSIMDBC64 xcodebuild/ReleaseSIMDBC64/run_vm_tests --ignore-unrecognized-flags UseDartApi
Took 0:00:00.897250

Short reproduction command (experimental):
    python tools/test.py -mrelease -asimdbc64 -t240 vm/cc/UseDartApi```

This test is now marked as Skip by a14d98ecf5475f7e0db248892656900a478c70dc. Can we find an owner for this crash and re-enable the test? /cc @peter-ahe-google @kasperl


4b normalization produces images with negative expression values

Describe the bug
Expression values on certain channels (48, 56, 69, 70) on certain FOVs on certain runs, were inverted after running normalization notebook. When the effected image were inverted again by multiplying by -1, the original image was restored (see example image attached).
Cohort size = 46 separate runs, ~650 individual FOVs
list of FOVs effected (image attached)
Expected behavior
No image should have negative expression values.
To Reproduce
Can provide the link to dataset

I'll leave this issue open for now, but it appears the discussion will be moving to #317.

This will be closed by #325


Fix deprecation warning

removes deprecation warning to prepare for compatibility with symfony 6

Thank you very much! This will be fixed already by #416.


Staxrip <IP_ADDRESS> MASSIVE delay from indexing to actually opening the file.

I am using the latest stable build of Staxrip. I use it for 4k HDR encodes. I always do manual source selection. I'm using FFvideosource for the 10-bit pipeline with avisynth. As soon as the program is done indexing the file, everything minimizes. It appears to be doing something in the background because if I wait 5 - 10 minutes it finally opens up in the program where I am able to use the UI and make my selections.
This DOES not happen in version  <IP_ADDRESS> at all. It's basically instant from index to UI using the exact same sources/files/settings/filter.
I am not sure what has changed from that version to this one. However, I tried both framework .exe's that are included. Reinstalled avisynth etc and it's the same problem regardless.
On top of that error above^^^
If I'm already encoding a 4k video, open another instance of StaxRip to do some work/add a job. Sometimes it will fully crash my computer. This did not happen on the previous version of staxrip neither. Unfortunately I didn't get any logs etc.

I know Stax76 made some changes to the code from what I previous had, Not 100% sure what all the changes were. Plus ffms2 Indexer is not in the best shape right now, It has some issues(Which is beyond my control). At times the indexer can run twice for some reason.

I could be wrong. But with avisynth to encode 4k hdr with 10 bit pipe ffms2 is the best option?
Also I went back and tested the older version as stated above again to make sure and the file indexed and opened up immediately
I'd be down to test stuff out if you make changes.


thanos.io: Make menu section collapsible.

It would be awesome if we could make our menu more compact. It's now very long due to more & more items. Worth to allow some dynamic behavior?
I mean this:

Another cool improvement for @prmsrswt if you have time :beers: I know you applied for GSoC, so don't worry, for GSoC the UI is 10000% more exciting as it is about Thanos microservices UIs :hugs:

This would be great. I think it shouldn't take much time to implement. And yeah, I have more than enough time now due to the current situation for better or worse.


Coverage: LuaFileSystem not found

I thought I had tried this earlier, but apparently not. More beautifully hacky tests.
I had to remove ".. lfs" since it would just result in a nil concatenation error, a silly oversight.

What.
Tests on my machine work, but on Travis they don't. I'll do some more investigating on this...

Alright, I figured out why. Baste (rightfully) caches modules its already loaded. Busted doesn't run tests synchronously, so the import("./fs") line outside the test is run before the test, caches, and doesn't error since it's already imported.

Coverage decreased (-3.3%) to 96.144% when pulling d7f65faebfa98cdbb3bed257d045beaef6fd8593 on Kampfkarren:fs_spec into 615de5e0c09808a83dfd0ac9ffa4addc76d6e019 on LPGhatguy:master.

That coverage failure tells me I have an outdated repo, I think. I'll work on it more when I can.

I have succumbed to my urge to not edit baste.lua. I'll probably send a PR to baste with my changes.
In essence, I just made makeImport public and made caches specific to each makeImport.

If ok is false, lfs will contain the error code; that shouldn't be a nil concatenation error. pcall returns the error as the second value if the call fails. The only case where it would be nil would be if require threw an empty error, and it doesn't.
I don't really care about the coverage of this particular branch all that much anyways, I don't think. It's not worth emulating a broken environment to see if this branch works.

I forgot that pcall returned the error when I was writing the code.
As for your last paragraph, I know it doesn't really matter but luacov doesn't support ignoring a line and not having 100% just irks me.
Should the PR be closed?

I think I'll close this, and then hold out for a LuaCov patch that lets us ignore lines 🙂


Remove graphiql from cloak dev

As agreed on #3188, we're discontinuing graphiql from cloak dev.
Fixes #3199
Signed-off-by: Marcos Lilljedahl<EMAIL_ADDRESS>

@aluzzardi please review. Thank you!

I'm closing this since it'll be addressed in #3901


When will the fp8 training code in the NVILA  be released?

The techinique part code is already released at https://github.com/NVlabs/COAT
The full scripts with NVILA will be released next week.

The techinique part code is already released at https://github.com/NVlabs/COAT
The full scripts with NVILA will be released next week.

ok ，thank you


commands/.../test/local,pkg/test: add image flag to test local

Description of the change: Adds a --image flag to the test local subcommand. Currently only supports 1 deployment with 1 container (we may add support for multi-deployment, multi-container manifests in the future, but it would also require more flags from the user to identify which image needs to be replaced).
Motivation for the change: See #746
Resolves #746
cc @awgreene

@AlexNPavel Can we add a one line explanation and example usage of this flag in the writing-e2e-tests doc?

@hasbro17 Added documentation


How start PACE in specific

Hi guys,
I'am working a project with and I need that PACE start in a  specific "div" and not in "body".
how can i do it?
regards and thanks you
PS: sorry for my english

insert you pace script and css inside div.
       <div class="mydiv" id="staticProgress">
            <script src="../plugins/paceflash/pace.js"></script>
            <link href="../plugins/paceflash/pace.css" rel="stylesheet" />
        </div>

Or set selector
elements: {
     selectors: ['body']
   },


Non-Responsive 'Book Table' Button Issue

@khushi-joshi-05
The 'Book Table' button is currently non-responsive. It should redirect users to a booking page, which is not happening.
I would like to contribute to this project (gssoc'24 contributor)

@bristiHalder you can start working on it, itstead of a complete new page, if it show a pop up page with say location it will be great.

@khushi-joshi-05
i would like to solve this button responsivity issue.(GSSoC`24 Contributer)


undefined error on Check availability

fixed

@sharkbit how can I archive this if I can't check it? What was the fix? I liked the reference to the fixed code in a previous Issue


vkcube, vkcubepp  should be modified to query/enable the portability extension

Due to the nature of VUID 04451, any existing app running on a portability platform (e.g., MoltenVK, gfx-rs) running validation will encounter new validation errors. To avoid the validation errors the application needs to query the existence of the extension and if present, enable the extension.
vkcube and vkcubepp should be modified to query/enable the portability extension.
See this portability issue for more information.
For reference, VUID 04451 states:

If the VK_KHR_portability_subset extension is included in pProperties of
vkEnumerateDeviceExtensionProperties, ppEnabledExtensions must include
"VK_KHR_portability_subset".

This was added in PR #479

This was added in PR #479


Python 3 compatibility

Why Python 3 compatibility? Why not?

I missed this PR come in. Fantastic--we really should move to py 3. Will review and test more tomorrow. Thanks!

@DanielVF is this PR to indented to be a move to Python3, or just updating our coding style to be ready for the eventual move?
I see runtime.txt was not updated, thus the question.

This was just intended to make the code able to run on both Python 2 and Python 3.  I wasn't actually flipping the runtime.

I'm okay if we decide to move to Python 3, but not excited about maintaining support for 2 versions of Python for something that only we use.

I agree with you on not having two supported platforms.
I think we should move to Python 3 now since Python 2 goes end-of-life in less then two years, and switching now is essentially.

Python 3 not a priority on the website at this time.


[language][vm] rewrite uleb serialization/deserialization and improve size checking

Summary
This introduces a few improvements to the serializer/deserializer:

Unified handling of uleb numbers.
Making it possible to explicitly specify a limit on each integer type, which will be enforced by both the serializer and deserializer. The limits used to be implied by the runtime type of the integer (u8, u16, etc.), which was inflexible and hard to reason about.

Test Plan
cargo test

this seems fine and it may help us to control those value.
I am not sure about all the indexes defined. A table max is a u16::max and I don't think we would ever want special treatment for each table.
However, having said so I am fine with whatever we want to get in and I would say let's get this quickly "in shape" for check in.
We can and will correct max values and the number and kind of values while we finish the spec.

@bors-libra r=dariorussi

:pushpin: Commit 547e823 has been approved by dariorussi

:hourglass: Testing commit 547e82321f84e524c6621c4c795069633efe0d60 with merge a7a24016f713ef00eadd74c908532b83db753f9c...

:broken_heart: Test failed - checks-actions_land_blocking_test

@bors-libra retry

:hourglass: Testing commit 547e82321f84e524c6621c4c795069633efe0d60 with merge 15279cb62db98e1b73c2382a8126575c926c73a8...

:sunny: Test successful - checks-actions_land_blocking_test, checks-circle_commit_workflow
Approved by: dariorussi
Pushing 15279cb62db98e1b73c2382a8126575c926c73a8 to master...


More global page events

Hello, I found phonon had 'pagecreated' and 'pageopened', but it hard to use for mvvm.
Could you support more events like framework7?

Framework7 looks like jQuery Mobile, really have more page events. Phonon have only this events from Core Navigator

I believe that fewer events leave more organized code. My humble opnion.

Yes, by now we have only this events, but you can fork project and add new events or changes :+1:

@zhengnz : you mean more events like pageclosing, pageclosed, on page animation.. etc?

@Qathom Yes, beacause I can use these event bind model or unbind model.

@zhengnz I added more global events! Please see: the documentation under the "global events section"


Make labeled function declaration renaming use as

Current master syntax:
let add = (:first x, :second y) => x + y;

Goal:
let add = (:first as x, :second as y) => x + y;

Make sure this doesn't clash with the other usages of as. The keyword will be a bit overloaded now.
No syntax opinion debates please, and don't advertise this too much. Right now, we need to focus on shipping this syntax to unblock people. Syntax bikeshedding are reserved for the next breaking change.

It did. You can try them in the playground! https://reasonml.github.io/try/?reason=DYUwLgBA5gTi4QLwQBQD8B2BDAtiCWAzhBgJRIB8EARABIjDAD2NEA1GyQNxA


Try to get the newest node configuration every poll interval when monitoring IP Address subnets

Description
Try to get the newest node configuration every poll interval when monitoring IP Address subnets.

Related issues/PRs
https://github.com/projectcalico/calico/issues/8700

Tests

Current k8s node and IP.

NAME                                STATUS   ROLES    AGE   VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                           KERNEL-VERSION                 CONTAINER-RUNTIME
hw-sks-test-logging-logging-m9c27   Ready    <none>   44h   v1.25.15   <IP_ADDRESS>   <none>        Rocky Linux 8.8 (Green Obsidian)   4.18.0-477.27.1.el8_8.x86_64   containerd://1.6.22

Modify the IP of k8s node by configuring static IP.

hw-sks-test-logging-logging-m9c27   Ready      <none>   44h   v1.25.15   <IP_ADDRESS>   <none>        Rocky Linux 8.8 (Green Obsidian)   4.18.0-477.27.1.el8_8.x86_64   containerd://1.6.22

monitor-addresses log show ip changed and updated

2024-04-18 15:32:05.741 [INFO][76] monitor-addresses/autodetection_methods.go 229: Including CIDR information from host interface. CIDR="<IP_ADDRESS>/16"
2024-04-18 15:32:05.742 [INFO][76] monitor-addresses/startup.go 579: Node IPv4 changed, will check for conflicts
2024-04-18 15:32:05.767 [WARNING][76] monitor-addresses/startup.go 1004: IPv4 address has changed. This could happen if there are multiple nodes with the same name. node="hw-sks-test-logging-logging-m9c27" original="<IP_ADDRESS>" updated="<IP_ADDRESS>"
2024-04-18 15:32:05.808 [INFO][76] monitor-addresses/startup.go 372: Updated node IP addresses
2024-04-18 15:33:05.864 [INFO][76] monitor-addresses/autodetection_methods.go 229: Including CIDR information from host interface. CIDR="<IP_ADDRESS>/16"

Todos

[x] Tests
[ ] Documentation
[ ] Release note

Release Note

TBD

Reminder for the reviewer
Make sure that this PR has the correct labels and milestone set.
Every PR needs one docs-* label.

docs-pr-required: This change requires a change to the documentation that has not been completed yet.
docs-completed: This change has all necessary documentation completed.
docs-not-required: This change has no user-facing impact and requires no docs.

Every PR needs one release-note-* label.

release-note-required: This PR has user-facing changes. Most PRs should have this label.
release-note-not-required: This PR has no user-facing changes.

Other optional labels:

cherry-pick-candidate: This PR should be cherry-picked to an earlier release. For bug fixes only.
needs-operator-pr: This PR is related to install and requires a corresponding change to the operator.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

/sem-approve

/sem-approve

/retest

Removing "merge-when-ready" label due to new commits

/sem-approve

@caseydavenport  Can this PR be cherry-picked to 3.26.x?

@Levi080513 I think that's fine, but I am not sure if / when we'll do another v3.26 patch release - we're about to release v3.29 which would put v3.26 largely out of schedule for patches.

Well, then I'll wait for release v3.29.


Quick Fix CharString to be String type.

Problem
When running chip-tool commands that have char * as a param, there is a crash reported as follows:
munmap_chunk(): invalid pointer. #11798
Change overview
Changed the char * param in arguments from CharString to String type.
Testing
Tested without change and reproduced issue.

Fast tracking as this is an update to a test tool


Serenity/JS 3.0: Missing npm reference in package serenity-js/web

On my attempts to migrate the Cumbumber WebdriverIO template I got the following when running npm install:
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE   package: '<EMAIL_ADDRESS>npm WARN EBADENGINE   required: { node: '^14 || ^16', npm: '^6 || ^7' },
npm WARN EBADENGINE   current: { node: 'v16.14.2', npm: '8.5.0' }
npm WARN EBADENGINE }

Is there an npm reference to version ^8 missing?:
https://github.com/serenity-js/serenity-js/blob/8bd28e99ef21ce0cf44e9c95e64761b0a89196fa/packages/web/package.json#L47

Fixed on main


Scoping filter in Workday to AD provisioning

Reference link: https://docs.microsoft.com/en-us/azure/active-directory/saas-apps/workday-inbound-tutorial#customizing-the-list-of-workday-user-attributes
Let's say we have join criteria from workday to AD based on username with sAMAccountName.
Lets assume contingent to employee movement we are having duplicate username in Workday. In that case we are planning to update contingent's user name with "KeyWord" so that we can filter it out of scope in Azure AD provisioning service. And new employee's user name will be joined with existing sAMAccountName.
Will this process required Full sync to complete or can it be maintained over a regular hourly sync.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 799c371a-a17e-08aa-ef56-86eef2395f7e
Version Independent ID: 1339cc93-712c-3a32-8321-dfd016a50ed4
Content: Tutorial: Configure Workday for automatic user provisioning with Azure Active Directory
Content Source: articles/active-directory/saas-apps/workday-inbound-tutorial.md
Service: active-directory
Sub-service: saas-app-tutorial
GitHub Login: @cmmdesai
Microsoft Alias: chmutali

@cmmdesai, kindly help us on this.

Thanks! I have assigned this to @cmmdesai as requested.

@AmitavaHazra I don't think this approach will work. Can you drop me a mail at<EMAIL_ADDRESS>so we can get on a call to discuss your integration scenario? We will be able to move faster that way.

@cmmdesai ,
I have sent mail to you. Lets have a call whenever you are available.

#please-close


DNS parser updates and right click updates

DNS parser:

added recognition for a bunch of query types, including UPDATE queries.
added an opcode field
fixed a bug with  queries not being recognized as DNS

Right click updates:

added %DATE%, which is the URL params for date or startTime/stopTime.  This allows a right click to link back to another Moloch search in the same time range.
added %FIELD%, which is the field being clicked on (e.g. ip.src).  This allows pivoting to a Moloch search in a new tab
added %EXPRESSION%, which is the current search expression.  This allows drilling into a Moloch search in a new tab.  E.g. linking to %EXPRESSION% && %FIELD% == %TEXT%

Example config.ini:
PIVOTIP=url:/?%DATE%&expression=ip+%3D%3D+%TEXT%;name:Pivot ip;category:ip
FIELDPIVOTIP=url:/?%DATE%&expression=%FIELD%+%3D%3D+%TEXT%;name:Pivot %FIELD%;category:ip
This isn't perfect because it produces a broken URL if the expression is blank:
FIELDDRILLIP=url:/?%DATE%&expression=%EXPRESSION%+%26%26+%FIELD%+%3D%3D+%TEXT%;name:Drill %FIELD%;category:ip
See attached example screenshot of pivoting from the IP in a DNS response to a search for the IP:

I committed some minor follow up changes and tests


HUBOT_JENKINS_{1-N}_URL

How is {1-N} defined?
HUBOT_JENKINS_1_URL

like that?
Then I would do:
@bot jenkins list 1

Correct, that is how {1-N} is defined.
So I'm 100% clear, you are suggesting an update to the #list command that allows it to only display a requested server's jobs by the supplied server index instead of it always outputting the results across all servers (though grouped).

Closing.


DROOLS-1434: moving public interfaces to kie-dmn-api module

@mswiderski @tarilabs This is the same PR @tarilabs sent, but with the minor changes I mentioned. I believe it is ready to merge.
Can I ask you please to fix jbpm with these changes and merge both PRs at once, to avoid breaking the build?
Thank you

I propose to close this PR in favour of #41 .

closing as requested


Add support for warnings that do not error endpoints

Add a new concept of warnings and non-critical services. Both
will not cause the template and JSON endpoint to response with a
500 status code but with a regular 200. This change is backwards
compatible because the existing warnings are treated as errors by
default.
Close #191

Codecov Report

Merging #193 into master will decrease coverage by 0.2%.
The diff coverage is 50%.

@@            Coverage Diff             @@
##           master     #193      +/-   ##
==========================================
- Coverage   73.92%   73.71%   -0.21%
==========================================
  Files          28       29       +1
  Lines         303      312       +9
==========================================
+ Hits          224      230       +6
- Misses         79       82       +3

Impacted Files
Coverage Δ

health_check/exceptions.py
100% <ø> (ø)
:arrow_up:

health_check/contrib/psutil/backends.py
0% <0%> (ø)
:arrow_up:

health_check/conf.py
0% <0%> (ø)

health_check/backends.py
100% <100%> (ø)
:arrow_up:

health_check/views.py
96.96% <88.88%> (-3.04%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 00a229d...d609b07. Read the comment docs.

Released in 3.6.0


Dev pytorch from forge

Description
The current dev branch cannot be published on conda-forge due to PyTorch not being installed from conda-forge but from the pytorch channel. Therefore, I changed the PyTorch installation channel from PyTorch to conda-forge in the CI/CD pipeline to test if the test succeeds with the new dependencies.
Todos

[ ] If the tests succeed, one has to adjust the conda-forge upload to use PyTorch from conda-forge.

Questions
No questions so far.
Status

[ ] Ready to go

In the first round, the conda setup failed.
I have no clear idea, why this breaks. Maybe, for every run, a cached environment is updated instead of a new one is created? Or mamba uses cached builds from previous downloads? Maybe, @dominiquesydow or @mbackenkoehler can help? My only guess is to rename the environments in the CI/CD pipeline and in devtools/test_env.yml. Let's see if that helps.

@Old-Shatterhand There is no caching of conda environments or packages.
Pasting the error from the raw logs here:
2023-06-06T09:04:25.6344822Z Could not solve for environment specs
2023-06-06T09:04:25.6345202Z The following packages are incompatible
2023-06-06T09:04:25.6345856Z ├─ mkl <2022  is requested and can be installed;
2023-06-06T09:04:25.6346344Z └─ pytorch 1.13.0  is uninstallable because there are no viable options
2023-06-06T09:04:25.6346750Z    ├─ pytorch 1.13.0 would require
2023-06-06T09:04:25.6347273Z    │  └─ mkl >=2022.2.1,<2023.0a0 , which conflicts with any installable versions previously reported;
2023-06-06T09:04:25.6347694Z    └─ pytorch 1.13.0 would require
2023-06-06T09:04:25.6348044Z       └─ __cuda  , which is missing on the system.


'coroutine' object has no attribute 'bot_invoice_url'

async def create_invoice() -> None:
invoice = await crypto.create_invoice(asset='TON', amount=1.5)
print(invoice.bot_invoice_url)

Which version of aiocryptopay is installed on your system? You can check it by running the command pip show aiocryptopay.


Add swagger docs for Rafiki Admin

https://issues.apache.org/jira/projects/SINGA/issues/SINGA-402

Merging with #48


Use get_context for subscriptions as well

Right now subscriptions just receive "websocket" in the context and not what you set using get_context
Description
I changed get_context to be able to receive both websocket and request. Based on the request, one of the two will be set
Hopefully, this change shouldn't break who is currently overwriting get_context
ToDo
I can update the documentation once we agree on what interface we like for get_context
Types of Changes

[x] Core
[x] Bugfix
[ ] New feature
[ ] Enhancement/optimization
[ ] Documentation

Issues Fixed or Closed by This PR
-- None
Checklist

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation. (?)
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] I have tested the changes and verified that they work and don't break anything (as well as I can manage).

Hi, thanks for contributing to Strawberry 🍓!
We noticed that this PR is missing a RELEASE.md file. We use that to automatically do releases here on GitHub and, most importantly, to PyPI!
So as soon as this PR is merged, a release will be made 🚀.
Here's an example of RELEASE.md:
Release type: patch

Description of the changes, ideally with some examples, if adding a new feature.

Release type can be one of patch, minor or major. We use semver,so make sure to pick the appropriate type. If in doubt feel free to ask :)


Console logs

When I use the plugin, it logs some tests results in the console:

passed the hide test
passed the event check
about to check
about to check

Is it normal? How can I disable this behaviour?
Thanks, and very useful package by the way!

Hey @ntopulos I would suggest to upgrade to 2.5.0 which has removed these and also has some other bug fixes: https://github.com/IamAdamJowett/angular-click-outside/releases/tag/v2.5.0
You could also upgrade via bower or npm if that is your choice of installation.


run command cannot identify file with a blank space - Windows syntax issue

Hi,
I have an issue with run command. When I write the below code, it works perfectly.
import rpa as r
r.init(True,False)

r.keyboard('[win]r')
r.wait(2)
r.keyboard('G:\\RPA_testing\\rpa_python\\production_planning\\email_attachment\\Purchase Order_202100045.pdf')
r.wait(2)
r.keyboard('[enter]')
r.close()

But when I am using the r.run, it shows below error message. How could I resolve this issue?
Note that there is a blank space between pdf file name (Purchase Order).
import rpa as r
r.init(True,False)
r.run('cmd /c start G:\\RPA_testing\\rpa_python\\production_planning\\email_attachment\\Purchase Order_202100045.pdf')
r.close()

Regards,
Ekram

This is Windows command prompt syntax issue, for path name with space, try using double quotes to see if it helps -
r.run('cmd /c start "G:\\RPA_testing\\rpa_python\\production_planning\\email_attachment\\Purchase Order_202100045.pdf"')

What error message did you get? I don't have a Windows laptop to test.

Hi,
When I run the script, it doesn't show any error message. At the same time, it doesn't open the pdf also. Here is the screenshot.

It worked.

Oh wonderful, I'll close the issue but let me know again if something crops up


We need test cases for PR #121

It seems like we forgot to add test cases for the stuff in PR #121. We need to add them.

And I'll take care of that.

With PR #218, #220, #221, and #223, this problem has been resolved.


Getting the symbol name of an error library or reason

(CC @tiran and @tniessen for CPython and Node, respectively. Please correct me if I've misunderstood what your projects are trying to do here!)
Given a value like SSL_R_THING_WENT_WRONG, ERR_reason_error_string returns a string like "thing went wrong". There doesn't seem to be any way to get the symbol name of an error reason.
I've noticed that both Node and CPython care about doing this, both because their public APIs involve exposing the symbolic name of the error. Node does it by uppercasing and replacing spaces with underscores.
https://github.com/nodejs/node/blob/v19.2.0/src/crypto/crypto_util.cc#L466-L476
This inverts the transform OpenSSL applies to generate the reason strings.
https://github.com/openssl/openssl/blob/openssl-3.0.7/util/mkerr.pl#L565
However, this is not documented and some errors in OpenSSL have manually adjusted strings. SSL_R_AT_LEAST_TLS_1_0_NEEDED_IN_FIPS_MODE's error string is "at least TLS 1.0 needed in FIPS mode". But this isn't universal; SSL_R_UNABLE_TO_FIND_ECDH_PARAMETERS is "unable to find ecdh parameters", not "unable to find ECDH parameters".
CPython takes a different approach. They maintain a table mapping error code to symbol name, seeded by parsing OpenSSL's header files:
https://github.com/python/cpython/blob/v3.11.0/Tools/ssl/make_ssl_data.py
https://github.com/python/cpython/blob/v3.11.0/Modules/_ssl_data_300.h
https://github.com/python/cpython/blob/v3.11.0/Modules/_ssl.c#L6022
https://github.com/python/cpython/blob/v3.11.0/Modules/_ssl.c#L458
Neither of these seems a particularly satisfying solution. The Node solution breaks on errors like SSL_R_AT_LEAST_TLS_1_0_NEEDED_IN_FIPS_MODE, while the CPython solution requires updating CPython source every time OpenSSL introduces a new error. Given two different projects have had to work around this already, perhaps OpenSSL should just provide a function with the appropriate semantics.
Perhaps ERR_reason_symbol_name and ERR_lib_symbol_name to return values like AT_LEAST_TLS_1_0_NEEDED_IN_FIPS_MODE and SSL for SSL_R_AT_LEAST_TLS_1_0_NEEDED_IN_FIPS_MODE and ERR_LIB_SSL, respectively?

Has there been any update on this?

Are those projects having similar problems with other error sets, such as errno?  After all, errno comes with a number of macros as well, like EPERM, EBADF, ERFKILL, and a number of other macros that are hiding in a plethora of errno.h files.
Technically, this is possible to do, but within limits, as engines and providers get the error library number dynamically (OpenSSL's built-in providers cheat in this regard).  And CPython paves the way, the way they do it is exactly what I could think of top of my head.
Should this end up in libcrypto / libssl?  Not sure...  entirely separate library for anyone to use if they want to?  That has ramifications that might not be very tasteful...

Unlike SSL & TLS versions, with evolving implementations, errno hasn't changed meaningfully in decades. Untrue of OpenSSL.
If you feel that "parsing .h files" is the public API for this, could you own and provide a .h -> structured data (json?) generator plus a test suite guaranteeing its output is complete and accurate? That'd at least make it official rather than hacks other projects so and continually expect to break upon new openssl releases.
(An actual API or some generated json that perhaps mkerr.pl or similar outputs may be simpler and easier to maintain.)

We do have a "similar output", which is what CPython looks at rather than .h files.
crypto/err/openssl.txt
It's not as complete as one might with, as it doesn't contain the library codes, but that could certainly be added.
So what I hear from you is that it would be desirable if this was a bit more complete, and exported / installed alongside the libraries.  And even better if it was exported in a more modern machine digestable form.
That about right?

@levitte Are you envisioning that CPython would dynamically find these files at runtime and parse them? Or that this would be a build-time process?
If runtime, isn't this just a less reliable and more cumbersome (relies on finding a random data file) version of the ERR_reason_symbol_name API?
If it's build-time, this doesn't resolve the problem with updates described in the bug report. Since OpenSSL aims to be ABI-stable (i.e. it is possible to update OpenSSL without recompiling applications), this seems particularly bad. It means CPython cannot effectively take advantage of OpenSSL's ABI stability because it'd require a recompile to report errors correctly.

Rehashing what @davidben said to ping this issue... From a CPython perspective: We don't want to find, load, and parse an SSL error data file at runtime. That is complex (how do we find it? what happens when it doesn't exist? how do we know it is the correct one to go with the library we're linked against... sometimes statically? etc.) and potentially slow (we would likely do it at _ssl module import time, slowing down startup, to avoid added complexity inside the error reporting path).
The proposed ERR_reason_symbol_name and ERR_lib_symbol_name APIs would suffice for CPython.

Ok, this is doable, but has its limits:

error lib and reason symbols for engines and providers can't reasonably be supported (it would require an expansion of how they are declared by engines and providers).
error reasons for the ERR_LIB_SYS reason table can't reasonably be supported, as they map system errors (errno numbers), and it's not reasonable to think that we'd hold a catalogue of all the E macros that exist on diverse operating systems.

So essentially, the proposed ERR_reason_symbol_name() and ERR_lib_symbol_name() can reasonably only work with error libs and reasons defined by crypto/err/openssl.txt.


Add automated releases using travis CI

we are currently running this release script manually: https://github.com/archilogic-com/3dio-js/blob/master/tasks/release.js
goal:
setup travis CI to run release script automatically (for master branch only!) when version number gets updated in a commit.
to do:

[ ] generate NPM auth token: https://docs.travis-ci.com/user/deployment/npm/
[ ] add travis CI config file .travis.yml to the repo (example: https://github.com/archilogic-com/3dio-website/blob/master/.travis.yml)
[ ] enable travis CI github service in: https://github.com/archilogic-com/3dio-js/settings/installations
[ ] add ENV variables for S3 and NPM in travis CI settings (ask tomas for the S3 keys)
[ ] modify release script to compare release version number from NPM to version number in package.json and continue only if they are different. Release script exit without error if version numbers are the same.
[ ] add step to release script preventing the script being run from repositories other than origin

@m-ke i though this one might be interesting to you? it is similar to jenkins and integrates nicely with github...


Add more comparison benchmarks

Hi there!
Your benchmarks look promising, but I would like to see how it compares to Thrift, gRPC and WASM.
These are some of the most popular alternatives for building RPC. I'd love to explore this for Tailcall as we are building a plugin API.

I think I can deliver a gRPC benchmark quickly, I have no experience with Thrift so if someone would like to support here I would be super happy.
@tusharmath I do not completely get what you mean with WASM. I know WebAssembly but unsure what you would like to compare here?

@elfenpiff Communication from Native (Rust) to a WASM module. While building a WASM based application you often need to communicate with the real world for example making and HTTP call or a DB query. Data needs to be sent from the native realm where the drivers are actually implemented to the application code which is typically in WASM. If the performance of iceoryx can beat this overhead of communicating with a WASM module, I think it will make it worthy candidate for building distributed applications.

@tusharmath it would be desirable to make iceoryx2 work in a WASM module but I think we would need to break out of the sandbox to make this work. I don't have experience with WASM, just reading about it now and then, but I think it will be quite challenging to solve that problem.


Uploading monaco editor to S3 should happen with the public-read permission

Right now they're private so no-one can use the Monaco Editor.

Done!


Make TextVectorization support pickling

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template
System information

TensorFlow version Tensorflow 2.2:
Are you willing to contribute it No:

Describe the feature and the current behavior/state.
Make TextVectorization support pickling, currently it does not support pickling:
import cloudpickle
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization
ptf = cloudpickle.dumps(TextVectorization())

Get error like Cannot convert a Tensor of dtype resource to a NumPy array.
Will this change the current api? How?
No
Who will benefit with this feature?
Support pickling will make it work with some distributed framework such as Horovod. Such framework usually require to pickle the user defined training function to remote and executing. So pickling is required.
Any Other info.
N/A

As mentioned here, pickling isn't natively supported in keras as it is generally not secure. Therefore I don't think this is actionable on Keras side, and overriding methods for tf.keras.layers.serialize is necessary. Thanks!