add cost and token logging in openai online and batching with litellm completion_cost

Changes:

expose metrics (i.e. cost and tokens) on the top-level in GenericResponse

Resolving Issues:

#146
#147

Related Issues:

#57
#74

Next steps:

Add cost info in viewer
Progress bar with cost estimates

Testing:

examples/poem.py on GPT 4o, GPT 4o-mini
examples/distill.py batch=True, on GPT 4o, GPT 4o-mini

Cost and token number look accurate.

Example output from GPT 4o-mini
{
  "response_message": {
    "poems_list": [
      "**A Day in the Life of a Wandering Cloud**\n\nUp in the sky, I drift and glide,  \nA wandering cloud, a nomad wide,  \nWith fluffy form and a heart so free,  \nI dance with the wind, a spirit at sea.\n\nAt dawn’s embrace, I take my flight,  \nPainted in hues of golden light,  \nI cradle the sun on my soft, white back,  \nAs shadows of night begin to slack.\n\nThrough fields of blue, I roam and muse,  \nOver mountains tall and rivers’ hues,  \nI wander afar, where the lilies sway,  \nAnd whisper secrets of the day.\n\nI coat the sky with a cottony brush,  \nKissing the earth in a gentle hush,  \nWith rain that drips like sweet lullabies,  \nGiving thirsting flowers a soft surprise.\n\nBy noon I’m lost in a sea of sun,  \nReflecting the joy of the world’s long run,  \nI scatter my shade o’er a sunbaked street,  \nWhile children look up, their laughter sweet.\n\nAs afternoon wanes, I’m painted in gray,  \nHeavy with thoughts of the end of the day,  \nI gather my rains and prepare to weep,  \nAs twilight whispers, I’m ready to sleep.\n\nRolling onwards, through dusk’s embrace,  \nI watch the stars in their velvet space,  \nMoonslight beckons, the night draws near,  \nA wandering cloud, with nothing to fear.  \n\nSo I drift through the night, a soft sight to spy,  \nA promise to rise with the day by and by,  \nAnd thus my tale ends, but only for now,  \nTomorrow’s adventure awaits this cloud!",
      "**The Journey of a Wandering Cloud**\n\nHigh above the world, where the eagles soar,  \nI’m a wandering cloud, forever wanting more,  \nA sailor of skies, with a shifting plan,  \nTracing the edges of nature’s span.\n\nIn the early light, I stretch and form,  \nA billowy shape, profoundly warm,  \nWith threads of silver and whispers of blue,  \nI sail the heavens, an ethereal view.\n\nThrough valleys and forests, o’er hills so green,  \nI peek down gently, the earth’s soft sheen,  \nI tickle the treetops, a playful caress,  \nWith a sprinkle of raindrops, I bless and press.\n\nBy noon’s bright blaze, I morph and swell,  \nAs children look up, they know me well,  \nI cast cool shadows on hot sandy shores,  \nA guardian spirit, the sky opens doors.\n\nBut storms will gather, a tempest brews,  \nWith rumbling thunder and darkened hues,  \nI gather my courage, embrace my design,  \nTo rain down blessings, a drenched divine.\n\nAs evening descends, I blush with the sun,  \nPainting the horizon—my day has been fun,  \nWith streaks of purple, and reds shining bright,  \nI whirl through the sky in the fading light.\n\nSo, when the stars emerge one by one,  \nI drift ever lightly, my journey not done,  \nFor tomorrow I’ll roam on a new summer's breeze,  \nAs a wandering cloud, I dance with such ease."
    ]
  },
  "response_errors": null,
  "raw_response": {
    "id": "chatcmpl-AVpkuxPgReKS8ZUfwyzc9WmtVxbZy",
    "object": "chat.completion",
    "created":<PHONE_NUMBER>,
    "model": "gpt-4o-mini-2024-07-18",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "{\"poems_list\":[\"**A Day in the Life of a Wandering Cloud**\\n\\nUp in the sky, I drift and glide,  \\nA wandering cloud, a nomad wide,  \\nWith fluffy form and a heart so free,  \\nI dance with the wind, a spirit at sea.\\n\\nAt dawn’s embrace, I take my flight,  \\nPainted in hues of golden light,  \\nI cradle the sun on my soft, white back,  \\nAs shadows of night begin to slack.\\n\\nThrough fields of blue, I roam and muse,  \\nOver mountains tall and rivers’ hues,  \\nI wander afar, where the lilies sway,  \\nAnd whisper secrets of the day.\\n\\nI coat the sky with a cottony brush,  \\nKissing the earth in a gentle hush,  \\nWith rain that drips like sweet lullabies,  \\nGiving thirsting flowers a soft surprise.\\n\\nBy noon I’m lost in a sea of sun,  \\nReflecting the joy of the world’s long run,  \\nI scatter my shade o’er a sunbaked street,  \\nWhile children look up, their laughter sweet.\\n\\nAs afternoon wanes, I’m painted in gray,  \\nHeavy with thoughts of the end of the day,  \\nI gather my rains and prepare to weep,  \\nAs twilight whispers, I’m ready to sleep.\\n\\nRolling onwards, through dusk’s embrace,  \\nI watch the stars in their velvet space,  \\nMoonslight beckons, the night draws near,  \\nA wandering cloud, with nothing to fear.  \\n\\nSo I drift through the night, a soft sight to spy,  \\nA promise to rise with the day by and by,  \\nAnd thus my tale ends, but only for now,  \\nTomorrow’s adventure awaits this cloud!\",\"**The Journey of a Wandering Cloud**\\n\\nHigh above the world, where the eagles soar,  \\nI’m a wandering cloud, forever wanting more,  \\nA sailor of skies, with a shifting plan,  \\nTracing the edges of nature’s span.\\n\\nIn the early light, I stretch and form,  \\nA billowy shape, profoundly warm,  \\nWith threads of silver and whispers of blue,  \\nI sail the heavens, an ethereal view.\\n\\nThrough valleys and forests, o’er hills so green,  \\nI peek down gently, the earth’s soft sheen,  \\nI tickle the treetops, a playful caress,  \\nWith a sprinkle of raindrops, I bless and press.\\n\\nBy noon’s bright blaze, I morph and swell,  \\nAs children look up, they know me well,  \\nI cast cool shadows on hot sandy shores,  \\nA guardian spirit, the sky opens doors.\\n\\nBut storms will gather, a tempest brews,  \\nWith rumbling thunder and darkened hues,  \\nI gather my courage, embrace my design,  \\nTo rain down blessings, a drenched divine.\\n\\nAs evening descends, I blush with the sun,  \\nPainting the horizon—my day has been fun,  \\nWith streaks of purple, and reds shining bright,  \\nI whirl through the sky in the fading light.\\n\\nSo, when the stars emerge one by one,  \\nI drift ever lightly, my journey not done,  \\nFor tomorrow I’ll roam on a new summer's breeze,  \\nAs a wandering cloud, I dance with such ease.\"]}",
          "refusal": null
        },
        "logprobs": null,
        "finish_reason": "stop"
      }
    ],
    "usage": {
      "prompt_tokens": 79,
      "completion_tokens": 794,
      "total_tokens": 873,
      "prompt_tokens_details": {
        "cached_tokens": 0,
        "audio_tokens": 0
      },
      "completion_tokens_details": {
        "reasoning_tokens": 0,
        "audio_tokens": 0,
        "accepted_prediction_tokens": 0,
        "rejected_prediction_tokens": 0
      }
    },
    "system_fingerprint": "fp_0705bf87c0"
  },
  "raw_request": {
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Write two poems about A Day in the Life of a Wandering Cloud."
      }
    ],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "output_schema",
        "schema": {
          "properties": {
            "poems_list": {
              "description": "A list of poems.",
              "items": {
                "type": "string"
              },
              "title": "Poems List",
              "type": "array"
            }
          },
          "required": [
            "poems_list"
          ],
          "title": "Poems",
          "type": "object"
        }
      }
    }
  },
  "generic_request": {
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Write two poems about A Day in the Life of a Wandering Cloud."
      }
    ],
    "response_format": {
      "properties": {
        "poems_list": {
          "description": "A list of poems.",
          "items": {
            "type": "string"
          },
          "title": "Poems List",
          "type": "array"
        }
      },
      "required": [
        "poems_list"
      ],
      "title": "Poems",
      "type": "object"
    },
    "original_row": {
      "topic": "A Day in the Life of a Wandering Cloud"
    },
    "original_row_idx": 7
  },
  "created_at": "2024-11-21 00:54:40.638421",
  "finished_at": "2024-11-21 00:54:46.237762",
  "token_usage": {
    "prompt_tokens": 79,
    "completion_tokens": 794,
    "total_tokens": 873
  },
  "response_cost": 0.00048824999999999997
}

(79 * 0.150 + 794 * 0.6) / 1e6 = 0.00048824999999999997 ✅
{
  "response_message": {
    "poems_list": [
      "### The Sun-Kissed Wanderer\n\n**At Dawn's Gentle Embrace:**  \nSprung from misty dreams of night,  \nA cloud awakens, soft and white,  \nWith the sun's first gentle beams,  \nStretches o'er the earth and streams.\n\nBlown by breezes, light and free,  \nIt drifts towards the open sea,  \nOver hills with morning dew,  \nBathed in sky's eternal blue.\n\n**The Midday Dance:**  \nOver grasslands, vast and wide,  \nWith the sun now high in the sky,  \nThe cloud takes on a cotton shape,  \nCasting cool shadows, sweet escape.\n\nOn its journey, whispering tales,  \nTo eagles soaring, as it sails,  \nLeaves glisten in its gentle wake,  \nAs rivers with its soft shade quake.  \n\n**Towards Dusk's Gentle Farewell:**\nAs the sun sinks low, turning gold  \nThe cloud crests mountains, strong and bold,  \nDraped in colors of fiery hue,  \nA tapestry of pink, orange, blue.\n\nIn twilight's arms, it finds its rest,  \nOn a mountaintop's airy nest.  \nThe wider world hushes to sleep,  \nAs stars ascend and shadows creep.\n\n### Whispering Wanderer’s Lullaby\n\n**The Morning Call:**  \nFrom dreams of moonlit realms afar,  \nEmerges at dawn a wandering star  \nIn a sea of skies where the chambered mist  \nKisses the world in a sunlit tryst.\n\nBreezes greet the cloud with cool caress,  \nIt glides with a din of quietness,  \nFloating past trees, sharing secrets, small,  \nLeaves and branches nod to its call.\n\n**The Noon Whisper:**  \nUnleashed by warmth, the cloud runs free,  \nIts shadow a ship upon the sea,  \nWith sailors below wishing voyages bright  \nBeneath its alabaster flight.\n\nPeeking through forests dense and wide,  \nIt weaves between the sun’s pride,  \nTossing travels of a thousand miles  \nIn form of playful, wispy smiles.\n\n**The Evening Embrace:**\nSunset waltzing with colors, bold,  \nTurns a weary cloud to shades of gold,  \nAs heavens paint its curtain call  \nWith wondrous fervor over all.\n\nNestled within the twilight hush,  \nOn mountain snow with glowing blush,  \nSlumber speaks in airy tunes,  \nBeneath the gaze of silvery moons."
    ]
  },
  "response_errors": null,
  "raw_response": {
    "id": "chatcmpl-AVpo5spkMtbKWdDSqpU6M4Asaz1Ab",
    "object": "chat.completion",
    "created":<PHONE_NUMBER>,
    "model": "gpt-4o-2024-08-06",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "{\"poems_list\":[\"### The Sun-Kissed Wanderer\\n\\n**At Dawn's Gentle Embrace:**  \\nSprung from misty dreams of night,  \\nA cloud awakens, soft and white,  \\nWith the sun's first gentle beams,  \\nStretches o'er the earth and streams.\\n\\nBlown by breezes, light and free,  \\nIt drifts towards the open sea,  \\nOver hills with morning dew,  \\nBathed in sky's eternal blue.\\n\\n**The Midday Dance:**  \\nOver grasslands, vast and wide,  \\nWith the sun now high in the sky,  \\nThe cloud takes on a cotton shape,  \\nCasting cool shadows, sweet escape.\\n\\nOn its journey, whispering tales,  \\nTo eagles soaring, as it sails,  \\nLeaves glisten in its gentle wake,  \\nAs rivers with its soft shade quake.  \\n\\n**Towards Dusk's Gentle Farewell:**\\nAs the sun sinks low, turning gold  \\nThe cloud crests mountains, strong and bold,  \\nDraped in colors of fiery hue,  \\nA tapestry of pink, orange, blue.\\n\\nIn twilight's arms, it finds its rest,  \\nOn a mountaintop's airy nest.  \\nThe wider world hushes to sleep,  \\nAs stars ascend and shadows creep.\\n\\n### Whispering Wanderer’s Lullaby\\n\\n**The Morning Call:**  \\nFrom dreams of moonlit realms afar,  \\nEmerges at dawn a wandering star  \\nIn a sea of skies where the chambered mist  \\nKisses the world in a sunlit tryst.\\n\\nBreezes greet the cloud with cool caress,  \\nIt glides with a din of quietness,  \\nFloating past trees, sharing secrets, small,  \\nLeaves and branches nod to its call.\\n\\n**The Noon Whisper:**  \\nUnleashed by warmth, the cloud runs free,  \\nIts shadow a ship upon the sea,  \\nWith sailors below wishing voyages bright  \\nBeneath its alabaster flight.\\n\\nPeeking through forests dense and wide,  \\nIt weaves between the sun’s pride,  \\nTossing travels of a thousand miles  \\nIn form of playful, wispy smiles.\\n\\n**The Evening Embrace:**\\nSunset waltzing with colors, bold,  \\nTurns a weary cloud to shades of gold,  \\nAs heavens paint its curtain call  \\nWith wondrous fervor over all.\\n\\nNestled within the twilight hush,  \\nOn mountain snow with glowing blush,  \\nSlumber speaks in airy tunes,  \\nBeneath the gaze of silvery moons.\"]}",
          "refusal": null
        },
        "logprobs": null,
        "finish_reason": "stop"
      }
    ],
    "usage": {
      "prompt_tokens": 79,
      "completion_tokens": 622,
      "total_tokens": 701,
      "prompt_tokens_details": {
        "cached_tokens": 0,
        "audio_tokens": 0
      },
      "completion_tokens_details": {
        "reasoning_tokens": 0,
        "audio_tokens": 0,
        "accepted_prediction_tokens": 0,
        "rejected_prediction_tokens": 0
      }
    },
    "system_fingerprint": "fp_7f6be3efb0"
  },
  "raw_request": {
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Write two poems about A Day in the Life of a Wandering Cloud."
      }
    ],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "output_schema",
        "schema": {
          "properties": {
            "poems_list": {
              "description": "A list of poems.",
              "items": {
                "type": "string"
              },
              "title": "Poems List",
              "type": "array"
            }
          },
          "required": [
            "poems_list"
          ],
          "title": "Poems",
          "type": "object"
        }
      }
    }
  },
  "generic_request": {
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Write two poems about A Day in the Life of a Wandering Cloud."
      }
    ],
    "response_format": {
      "properties": {
        "poems_list": {
          "description": "A list of poems.",
          "items": {
            "type": "string"
          },
          "title": "Poems List",
          "type": "array"
        }
      },
      "required": [
        "poems_list"
      ],
      "title": "Poems",
      "type": "object"
    },
    "original_row": {
      "topic": "A Day in the Life of a Wandering Cloud"
    },
    "original_row_idx": 7
  },
  "created_at": "2024-11-21 00:57:57.274561",
  "finished_at": "2024-11-21 00:58:13.849180",
  "token_usage": {
    "prompt_tokens": 79,
    "completion_tokens": 622,
    "total_tokens": 701
  },
  "response_cost": 0.0064175000000000005
}

(79 * 2.50 + 622 * 10.00) / 1e6 = 0.0064175000000000005 ✅


ftl-runner cache has almost 200,000 cached runners

When pairing with @mistermoe we saw that he was getting some permission denied on some cached runners so we went to delete the caches and noticed a ton of runner* folders.
I then took a look at my machine and saw ~185,000 similar folders.

We had an issue where if your laptop hibernated runner creation could go into a loop with a cancelled context. Adding the max retries stopped the loop, and https://github.com/TBD54566975/ftl/pull/3239 should work around the context issue.


Is it support video call?

Hi,
No it does not support Video - this project implements methods from the Twilio Client SDK, and the Twilio Video SDK is a separate project - I'm not familiar with any Phonegap plugins for Twilio Video.
Thanks!


feat: add eligibility requirements

This PR clarifies the eligibility requirements for getting access to the index as discussed here: https://algolia.slack.com/archives/C2WQLD93Q/p1632350387498900?thread_ts=1632329402.484700&cid=C2WQLD93Q.

Thanks for that @Meschreiber 🙏🏻


Not sure how this could occur

I'll commit this to main soon, but I made it so that when you click on an image in the forums, it gets enlarged. Flask doesn't seem to be rendering the page correctly.

I fixed this in my latest commit. Basically, the problem was with with '_base.html' . So I just removed it since most of that wasn't needed anyways for the images.

I fixed this in my latest commit. Basically, the problem was with with '_base.html' . So I just removed it since most of that wasn't needed anyways for the images.

Did you remove the entire file???

Oops, sorry lol. Should I close this issue, or is there still a problem?

Idk, haven't tested it

Its already in the code.


Suggested Readme improvements and guide for Phoenix integration

My environment is Linux Mint 17.3 (Trusty Tahr) with version 19.3 of Erlang/OTP, Elixir 1.4.2, Rustc 1.17.0.
I noticed a few inconsistences with the guide and readme and here is how I resolved them:
The README.md generated with the rustler.new command has the following code:
defmodule ModuleName
require Rustler

@on_load :load_nif
def load_nif do
    Rustler.load_nif("projectname")
end
...
end

This code throws an Elixir error as-is.  The code at https://hexdocs.pm/rustler/basics.html does not match this autogenerated README. Neither of them work as-is.

== Compilation error on file lib/module_name.ex ==
** (MatchError) no match of right hand side value: {:error, :on_load_failure}
(stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
(elixir) lib/kernel/parallel_compiler.ex:117: anonymous fn/4 in Kernel.ParallelCompiler.spawn_compilers/1

After some experimentation, here's what I got to work:
defmodule ModuleName
    use Rustler, otp_app: :appName, crate: :cratename

    :load_nif  #Removed @on_load, but including :load_nif is important! Causes a segfault indeterminantly!
    def load_nif do
        Rustler.load_nif("nifname")
    end

    # When your NIF is loaded, it will override this function.
    def add(_a, _b), do: throw :nif_not_loaded
end

As noted in the comment above, I got a really strange segmentation fault when I ran this without the :load_nif code with Phoenix. It worked, consistently, in an elixir/mix project outside of Phoenix, but as soon as I had the exact same source in the /native/ directory of my Phoenix project, it would sometimes segfault and sometimes run. I could enter iex -S mix, get it to run, exit, then re-run iex -S mix, and then it would segmentation fault.
As for a guide on including with Phoenix:
This may not be obvious for some people, but there is already a "compilers:" block in the project function. Including an additional compilers block, like I did because I didn't know any better, results in Rustler and Phoenix looking in the _build/dev/lib/project/priv directory and being unable to find the Rust crates.
def project do
compilers: [:rustler, :phoenix, :gettext ] ++ Mix.compilers, #<---- How important is the order here?!
rustler_crates: rustler_crates() #<--- does this position matter?

I don't think the order matters here but I could be wrong. It would be clear to explicitly state that.
While trying to figure out my problem I played around with the ordering some and I'm not convinced that the ordering of the atoms and the listing of the crates doesn't matter yet.
I hope this is helpful as this is a great project and I see lots of potential with Rust + Phoenix.

Thanks for the feedback!
I will update the documentation to make some of these things more clear.
I am really curious about the segfault you are getting. This is not something that should happen, and I would be very interested in figuring out what is going on here.

I went through and tried to replicate the segfault but I was not able to with a clean install. If I find it again I will narrow it down to a particular issue now that I know how to get everything working.
I also created a tutorial on how to get rustler up and running with phoenix here: https://github.com/cgregfreeman/phoenix_rustler_integration
Feel free to copy and edit or use as needed.

I changed the documentation around a bit in the generated README.

I implemented working example here - https://github.com/korczis/rustler-test


Generate varlink API documentation automatically

Using varlink's idl parser, we generate API documentation for the podman
API relying on the .varlink file as the source.
Signed-off-by: baude<EMAIL_ADDRESS>

@baude looks like a rebase is needed.

bot, retest this please

This LGTM once tests are green

bot, retest this please

@rh-atomic-bot r+


Bug: Incorrect list item creation under certain circumstances.

Hi,
https://gitbookio.github.io/slate-edit-list/
If you place your cursor at the end of this line for example Third item in the list, with a nested list and press enter a new child item is created however it looks like another parent list is created at the same level of what use to be the first child First item in the nested list.
I would presume what should occur is that it would just create a new list item at the same level of the item enter was pressed on.
On another note, is it possible to disable or prevent certain list edit behaviour occuring. For example i would like to stop the block splitting behaviour being possible (shift + enter).
Thanks

Also ran into this today as well. Regarding your sidenote, @deep-c, comment out https://github.com/GitbookIO/slate-edit-list/blob/95d758a3e0a29e42cc3b23d768fc182eedb53597/lib/onEnter.js#L17-L19 to remove the shift-enter functionality (I'm assuming here you want shift-enter to create a new item)

The bug should be fixed.
I opened an issue #59 concerning behavior options


Does Embark support solidity libraries?

Deployment of a contract which uses solidity library fails with 'out of gas exception'

+1
I'm new to Solidity but this simple test program is not working for me. I tried it on Remix and it showed no errors.
pragma solidity ^0.4.7;

library L {

  function add(uint _a, uint _b) returns (uint _c) {
    return _a + _b;
  }

}

contract Test {

  function testAdd() constant returns (uint _result) {
    return L.add(1, 1);
  }

}

I would copy and paste the error but the console won't allow me to select the text. But as the OP said, it starts out as:
VM Exception while processing transaction: out of gas
  at /usr/local/lib/node_modules/embark/lib/contracts/deploy.js:76:27

I tried increasing default.gas in contracts.json and it still failed.

@antonina-cherednichenko @jake17007 looks like there is an issue with the library deployment, it's being fixed as we speak and to be released on version 2.6.0

Any idea when this fix will be released, or a workaround? It's currently quite blocking

@Qkyrie We're pointing for next monday however you can use it right now by cloning the develop branch then running npm link (see here for any changes you might need to do in your contracts config)

@Qkyrie @antonina-cherednichenko @jake17007 @atfornes If you need help using develop let me know or drop by on gitter
The 2.6.0 release is not quite there yet, I might release this fix as a 2.5.2 patch instead this week.

Released 2.5.2 with the library fixes
Note if using an existing DApp: to add library support with embark blockchain you need to remove your embark data dir rm -rf .embark/development and add to your genesis block the config signalling homestead at config/development:
 "config": {
      "homesteadBlock": 1
    },

example config file here


CPI: Certain characters (e.g. 'ä', 'ö') are mangled with DeepPlain2XMLConverter

It seems that at least some characters (ä, Ä, ö, Ö) get mangled in DeepPlain2XMLConverter -conversion on CPI.
The scenario is from UTF-8 to UTF-8, so there should be no other sources of charset corruption.
The conversion script is simply;
import com.sap.gateway.ip.core.customdev.util.Message
import com.equalize.cpi.converter.FormatConversionBean

def Message processData(Message message) {
	def fcb = new FormatConversionBean(message.exchange, message.getProperties())
	def output = fcb.convert()

	message.setBody(output)
	return message
}

"Yrjö Kyrölä" converts into "Yrj�� Kyr��l��"
Might be related to issue "Encoding of umlaut character corrupted during XML to JSON conversion #10"

I'll create a short example iFlow shortly, there's a bunch of irrelevant process steps in my actual scenario.

I'll create a short example iFlow shortly, there's a bunch of irrelevant process steps in my actual scenario.

Hi @kepatron, thanks for providing all the details. I managed to reproduce the issue and found out where was the bug. It wasn't particularly obvious when working locally as my environment is set to use UTF-8 for the encoding. But somehow, on the CPI tenant it uses a different default file encoding, and that caused the issue. I've updated the particular code so that is uses specific encoding, rather that relies on the platform default encoding.
The fix is available in release 2.2.1 (https://github.com/engswee/equalize-cpi-converter/releases/tag/2.2.1), so if you use that, you should not need a workaround on the IFlow.

Hi @kepatron, thanks for providing all the details. I managed to reproduce the issue and found out where was the bug. It wasn't particularly obvious when working locally as my environment is set to use UTF-8 for the encoding. But somehow, on the CPI tenant it uses a different default file encoding, and that caused the issue. I've updated the particular code so that is uses specific encoding, rather that relies on the platform default encoding.
The fix is available in release 2.2.1 (https://github.com/engswee/equalize-cpi-converter/releases/tag/2.2.1), so if you use that, you should not need a workaround on the IFlow.


Two issues with underlying IRS data formats

First, I wanted to thank the authors for providing this public good!
For users, I wanted to note two potential issues with the data, according to my understanding of it.

Unlike for most files, some inflow files do not properly format leading zeros for states and counties. I noticed this for the 1998 inflow files for WV and PA. It affects the final dataset in that it generates erroneously coded county codes.

library(data.table)
dt <- data.table(read.csv('county_migration_data.csv'))
print(dt[dt$destination<1000 | dt$origin<1000,])

     origin destination X1990 X1991 X1992 X1993 X1994 X1995 X1996 X1997 X1998 X1999 X2000 X2001 X2002 X2003 X2004 X2005 X2006 X2007 X2008 X2009 X2010
  1:    101       42101     0     0     0     0     0     0     0     0    47     0     0     0     0     0     0     0     0     0     0     0     0
  2:    101        4217     0     0     0     0     0     0     0     0    22     0     0     0     0     0     0     0     0     0     0     0     0
  3:    101        4229     0     0     0     0     0     0     0     0    11     0     0     0     0     0     0     0     0     0     0     0     0
  4:    101         423     0     0     0     0     0     0     0     0    28     0     0     0     0     0     0     0     0     0     0     0     0
  5:    101        4245     0     0     0     0     0     0     0     0    25     0     0     0     0     0     0     0     0     0     0     0     0
 ---
661:   5491         541     0     0     0     0     0     0     0     0    26     0     0     0     0     0     0     0     0     0     0     0     0
662:   5497         541     0     0     0     0     0     0     0     0    26     0     0     0     0     0     0     0     0     0     0     0     0
663:   5497         547     0     0     0     0     0     0     0     0    18     0     0     0     0     0     0     0     0     0     0     0     0
664:   5525         423     0     0     0     0     0     0     0     0    27     0     0     0     0     0     0     0     0     0     0     0     0
665:   5579         423     0     0     0     0     0     0     0     0    30     0     0     0     0     0     0     0     0     0     0     0     0

Judging from the aggregate counts in the final dataset across years, there seems to be some leakage in the 1996 bucket of censored movers '99999'.

yearCols <- colnames(dt)[substr(colnames(dt),1,1)=='X']
years    <- as.integer(substr(yearCols,2,5))

par(mfrow=c(2,2))
plot(years, colSums(dt[,                          ..yearCols]), type='l',ylab='Total')
plot(years, colSums(dt[dt$destination==dt$origin, ..yearCols]), type='l',ylab='Non-Movers')
plot(years, colSums(dt[dt$destination=='99999',   ..yearCols]), type='l',ylab='Censored Movers')
plot(years, colSums(dt[dt$destination!='99999'&
                       dt$destination!=dt$origin, ..yearCols]), type='l',ylab='Non-Censored Movers')

Thanks! I'll take a look at the 1998 datafile next week. Thanks for letting me know!

Mathew Hauer, PhD
Assistant Professor, Department of Sociology
Center for Demography and Population Health
605 Bellamy Building, 113 Collegiate Loop
Florida State University, Tallahassee, FL, 32306
850-644-7103

From: Andrés Shahidinejad<EMAIL_ADDRESS>Sent: Friday, June 14, 2019 2:15 PM
To: mathewhauer/irs-migration-replication
Cc: Subscribed
Subject: [mathewhauer/irs-migration-replication] Two issues with underlying IRS data formats (#1)
First, I wanted to thank the authors for providing this public good!
For users, I wanted to note two potential issues with the data, according to my understanding of it.

Unlike for most files, some inflow files do not properly format leading zeros for states and counties. I noticed this for the 1998 inflow files for WV and PA. It affects the final dataset in that it generates erroneously coded county codes.

library(data.table)
dt <- data.table(read.csv('county_migration_data.csv'))
print(dt[dt$destination<1000 | dt$origin<1000,])
 origin destination X1990 X1991 X1992 X1993 X1994 X1995 X1996 X1997 X1998 X1999 X2000 X2001 X2002 X2003 X2004 X2005 X2006 X2007 X2008 X2009 X2010

1:    101       42101     0     0     0     0     0     0     0     0    47     0     0     0     0     0     0     0     0     0     0     0     0
2:    101        4217     0     0     0     0     0     0     0     0    22     0     0     0     0     0     0     0     0     0     0     0     0
3:    101        4229     0     0     0     0     0     0     0     0    11     0     0     0     0     0     0     0     0     0     0     0     0
4:    101         423     0     0     0     0     0     0     0     0    28     0     0     0     0     0     0     0     0     0     0     0     0
5:    101        4245     0     0     0     0     0     0     0     0    25     0     0     0     0     0     0     0     0     0     0     0     0
661:   5491         541     0     0     0     0     0     0     0     0    26     0     0     0     0     0     0     0     0     0     0     0     0
662:   5497         541     0     0     0     0     0     0     0     0    26     0     0     0     0     0     0     0     0     0     0     0     0
663:   5497         547     0     0     0     0     0     0     0     0    18     0     0     0     0     0     0     0     0     0     0     0     0
664:   5525         423     0     0     0     0     0     0     0     0    27     0     0     0     0     0     0     0     0     0     0     0     0
665:   5579         423     0     0     0     0     0     0     0     0    30     0     0     0     0     0     0     0     0     0     0     0     0

Judging from the aggregate counts in the final dataset across years, there seems to be some leakage in the 1996 bucket of censored movers '99999'.

yearCols <- colnames(dt)[substr(colnames(dt),1,1)=='X']
years    <- as.integer(substr(yearCols,2,5))
par(mfrow=c(2,2))
plot(years, colSums(dt[,                          ..yearCols]), type='l',ylab='Total')
plot(years, colSums(dt[dt$destination==dt$origin, ..yearCols]), type='l',ylab='Non-Movers')
plot(years, colSums(dt[dt$destination=='99999',   ..yearCols]), type='l',ylab='Censored Movers')
plot(years, colSums(dt[dt$destination!='99999'&
dt$destination!=dt$origin, ..yearCols]), type='l',ylab='Non-Censored Movers')
[image]https://urldefense.proofpoint.com/v2/url?u=https-3A__user-2Dimages.githubusercontent.com_11236348_59529027-2D81cdb180-2D8ea5-2D11e9-2D8ec9-2D08dbc772d748.png&d=DwMCaQ&c=HPMtquzZjKY31rtkyGRFnQ&r=4dsMboKrDZCTWM4zk0ZTRw&m=LDnWzLV2rIKw9-dMT1luaF25hhj6ohIkgkNgZqXOpEc&s=rGfp1kumttenbp9O-cTuCeCf3W4QKfNZ4iNmHMTca64&e=
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mathewhauer_irs-2Dmigration-2Dreplication_issues_1-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAD7TDMJWFG7SZCGDLEOETGDP2PN5DA5CNFSM4HYL2X7KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GZTW7WQ&d=DwMCaQ&c=HPMtquzZjKY31rtkyGRFnQ&r=4dsMboKrDZCTWM4zk0ZTRw&m=LDnWzLV2rIKw9-dMT1luaF25hhj6ohIkgkNgZqXOpEc&s=V5k6OKQSRZIX-PMTdEGdw2oNpBl2FbCQ-yHmKXgU1WQ&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AD7TDMMDGFNYMTLT3KPUBP3P2PN5DANCNFSM4HYL2X7A&d=DwMCaQ&c=HPMtquzZjKY31rtkyGRFnQ&r=4dsMboKrDZCTWM4zk0ZTRw&m=LDnWzLV2rIKw9-dMT1luaF25hhj6ohIkgkNgZqXOpEc&s=8KzVRjNk25En6xuowtpGRbS0IrPKv8IxUfZV3XvuiE8&e=.

So I found the problems and fixed them. There were two separate problems in the IRS migration data themselves.

Regarding the origin's/destination's with odd FIPS codes. For some reason, the PA and WV excel files provided by the IRS do not explicitly declare the state and county variables as characters.

Notice the county column starts with a '0' instead of a '000'.
I explicitly declared those columns as characters and used str_pad() to ensure the leading zeroes are included.

Regarding the leakage in the 1996 bucket. This is, again, a weird oddity in the IRS migration data. For some unknown reason, the IRS out-migration files randomly capitalize the filenames for some states and only for the out-migration files.

When pulling the files into R, the code now declares ignore.case=T and the files are all brought in correctly.
dt <- data.table(read.csv('DATA-PROCESSED/county_migration_data.csv'))
print(dt[dt$destination<1000 | dt$origin<1000,])

> Empty data.table (0 rows and 23 cols): origin,destination,X1990,X1991,X1992,X1993...

Shows there are no origins with less than 5 digits. Thus, the problem with PA and WV in 1998 should now be fixed.
yearCols <- colnames(dt)[substr(colnames(dt),1,1)=='X']
years    <- as.integer(substr(yearCols,2,5))

par(mfrow=c(2,2))
plot(years, colSums(dt[,                          ..yearCols]), type='l',ylab='Total')
plot(years, colSums(dt[dt$destination==dt$origin, ..yearCols]), type='l',ylab='Non-Movers')
plot(years, colSums(dt[dt$destination=='99999',   ..yearCols]), type='l',ylab='Censored Movers')
plot(years, colSums(dt[dt$destination!='99999'&
                         dt$destination!=dt$origin, ..yearCols]), type='l',ylab='Non-Censored Movers')


Add missing CRUD endpoints

Some resources are missing certain CRUD endpoints. This issue tracks which have been implemented and which still need to be added.

[x] Additional requirements

[x] update
[x] delete

[x] Plant communities

[x] delete

[x] Plant community actions

[x] update
[x] delete

[x] Indicator plants

[x] update
[x] delete

[x] Monitoring areas

[x] update
[x] delete

Added by #77


More use of returns instead of var/break

Similar to
https://github.com/optimizely/node-sdk/pull/61
This removes var placeholders and breaks in loops with direct return statements.

Coverage decreased (-0.004%) to 98.939% when pulling eb227173d74edfcf73ab0d3399fbc8543b0f61ba on avindra:fix/easier-loop-return into 91ff9677ce42ba8ed45668b0863906e933e73f80 on optimizely:master.

Sorry about the coverage loss :smile:

Coverage decreased (-0.003%) to 98.925% when pulling 51906343c9b955ae5fe6eb77d62e212b2ecec3ed on avindra:fix/easier-loop-return into 79e80e76ff517837fe45aa7bee3f902033da343d on optimizely:master.


Updated after hook to make sure it executes even if no of retries is 0

https://thinkneardev.atlassian.net/browse/ES-1118

Make sure you bump up the minor version number, so this change gets picked up.

Bumped up the version number. Ready for re-review.


Add Mechanism to prevent adding new unstable flaky to test

Is your feature request related to a problem? Please describe.
Recently, We observe more flakiness on our tests because of newly created tests or updates on existing tests, like the following 2 examples:
https://github.com/opensearch-project/OpenSearch/issues/5189
https://github.com/opensearch-project/OpenSearch/issues/5031
Describe the solution you'd like
I am looking for a mechanism to avoid this from happening again, this could be by manual or an automated process, like maintainers should not approve PR that has new test without providing 100+ successful run with no failures.
Looking to for better alternatives

@pranikum @imRishN @xuezhou25
Would like to know your thoughts on this?

This PR in the Gradle test-retry plugin is an interesting idea. The general approach is to identify tests that are added or changed as a part of a commit, then re-run those tests many times as a part of the build. The idea is to catch flakiness immediately so the developer that already has the context can fix them right away. This functionality doesn't exist in the test-retry plugin yet but it could be added.

@andrross a bit different approach is taken by Jenkins (if I am not mistaken): the tests diff is taken from JUnit XML reports (we also have them). So basically the idea would be:

fetch the reports from upstream branch (main, 2.x, ..)
compare with the report for current pull request (there could be new test cases in existing suites or completely new test suites)

The question is: when should be run such check? We probably should not be running it on each change in pull request (however it could be useful sometimes), but certainly before the pull request gets merged.

I propose new tests be rerun with N random seeds as part of gradle check (or a new gradle check for new tests).

While I do think this is an interesting idea, our quick approach of surfacing flakiness directly in a PR comment (#5200) seems to be working well for surfacing flakiness. It does require manual work from authors and reviewers to notice the failure, create an issue, and (for recently introduced flakiness) notify the original author, but I do believe this is working reasonably well now. I'm inclined to close this issue because I don't think anybody is going to take up this work in the near term to improve the automation here. @dblock @reta what do you think?

Let's close.
I'd a lot of things, like a Jenkins plugin that manages flaky test GitHub issues, but maybe we don't need to do that much engineering ;)


LAYOUTS-651 Show a modal error popup when reloading a block fails

Needs sync with @markosandalj in order not to conflict with his PR

@bcafuk Please rebase to resolve conflicts with master.

@emodric Rebased onto master.

Thanks @bcafuk !


chore: (IAC-1306) Add Bug Report Form

Changes
Add bug report form

This looks good @jarpat, wondering if you know of any way for us to hide/remove the option for creating a blank issue when folks are presented with both the "Get Started" green button that uses this template and the blue "Open a blank issue" which does not. Unfortunately, the blue link is what people end up using in my experience which is not ideal for us.

@dhoucgitter - like this<EMAIL_ADDRESS>

@dhoucgitter & @sayeun I'll take a look at that link

@sayeun That looks like it works to eliminate the "blank issues", a small change is needed, the extension needs to be yml rather than yaml.
I'll also create add a generic feature request issue based on the default one from GitHub since bugs are not the only thing that users will report. I'll make the additons and ask for reviews again.

@sayeun, @dhoucgitter, @thpang, @riragh this is ready for re-review. Added the suggestion to remove the "blank issues" option so we have more structured issues from users. Also added a generic feature request template.
Tried it out in my test repo, it will render like this:


[Feature][Transform] Add LLM transform

Purpose of this pull request
Add new LLM transform. Current support OpenAI api.
Leverage the power of a large language model (LLM) to process data by sending it to the LLM and receiving the
generated results. Utilize the LLM's capabilities to label, clean, enrich data, perform data inference, and
more.

Does this PR introduce any user-facing change?
no

How was this patch tested?
add new test and tested in local with own openai api key.

Check list

[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[ ] Update the release-note.

The current version processes data one by one, so its efficiency is relatively slow. Once transform supports generating multi-row data, it will be adapted in llm transform and theoretically the speed will be much faster.

Similar tools can be found at https://github.com/refuel-ai/autolabel

This is an experimental feature and can be discussed in the weekly meeting.


Project Comments

Describe the changes you have made in this PR -

Add threads endpoints.

close thread /api/v1/threads/:id/close
reopen thread /api/v1/threads/:id/reopen
subscribe thread /api/v1/threads/:id/subscribe
unsubscribe thread /api/v1/threads/:id/unsubscribe

Add comments endpoints

GET thread's comments /api/v1/threads/:thread_id/comments
POST new comment /api/v1/threads/:thread_id/comments
UPDATE comment /api/v1/comments/:id
Delete, Undelete, Upvote, Downvote, Unvote comment.

Format routes.rb.

Cleanup base_controller.rb.

Add extensive specs for threads & comments API.

Note: Please check Allow edits from maintainers. if you would like us to assist in the PR.

Coverage decreased (-5.7%) to 81.655% when pulling f200caffe5a41afb2e86eaf28b0d822e7841a32e on Nitish145:project_comments into 67e56093bf05c80331512e42dd891c613f17390e on CircuitVerse:master.

@tachyons please review.. onto adding tests..

@tachyons please have a look, done with adding specs..

@tachyons

Added checks for :index action..

If project is "Public", no need to be authenticated to fetch comments.
If project is not "Public", user needs to be authenticated as the author to fetch comments or security_transgression.

please check @tachyons ..

@tachyons


Why do I have to install openpyxl when I am using calamine as a pandas read_excel engine?

Example:
wr.s3.read_excel(file, sheet_name=0, engine="calamine")
Here I am using calamine as the engine to open an excel file. This is fully supported by pandas.
But, awswranger is throwing an error telling me that I have to install openpyxl, even tho I do not want to use it.
If I understand correctly, awswrangler is not using openpyxl but just delegating the call to pd.read_excel. If yes, why are you checking for this library? How can I know which engine is actually used?
If no, it would be better to let the user know that a different engine cannot be selected.

Thanks for raising this. I agree, not sure why openpyxl should be enforced

Thanks for the quick fix!


How to use detekt-formatting with precompiled binary in release?

There is the --plugins flag to add extensions:
detekt --plugins detekt-formatting/build/libs/detekt-formatting-1.11.0-RC1.jar  --input detekt-core/

Where can I find this jar?

You can download it from bintray: https://bintray.com/arturbosch/code-analysis/detekt/1.11.0-RC1
Specifically the mentioned jar is this one:
https://bintray.com/arturbosch/code-analysis/download_file?file_path=io%2Fgitlab%2Farturbosch%2Fdetekt%2Fdetekt-formatting%2F1.11.0-RC1%2Fdetekt-formatting-1.11.0-RC1.jar
Please close the issue if your issue is solved 👍

Thanks though I expect it should be released alongside the main cli.

Hmm I am not able to find 1.14 artifacts...

Hey @Mygod,
we moved to maven central (https://mvnrepository.com/artifact/io.gitlab.arturbosch.detekt/detekt-formatting/1.15.0-RC1) due to unstable releases when uploading to many files to Bintray.


[IDEA] Report/Suggestionbox cogs

I think a cool feature to add, would be for suggestion and reports, to give them an ID#, and staff that recieve these suggestions/reports, can respond directly to the user with the given ID, through the actual bot.
Example:
User A submits a report, with a report # of 1.
Then the staff team, can do [p]respond {ID#} {response}
This way, the report/suggestion can be replied too directly, straight through the bot.
Feel free to comment below and questions, or need a better explanation.

I'd like to make this happen, but my current priority is fixing any major bugs found and having all of the code ready for the release of Red v3. Good news is I am almost done with the latter.


Getting Started with Core Data Tutorial

原文链接：http://www.raywenderlich.com/115695/getting-started-with-core-data-tutorial
iOS9 Core Data

@MrLoong 认领


1.0.3

I'm getting ready for a 1.0.3 release. Anything anyone wants to see in 1.0.3? This realease is mainly for new features of #104 and #102.

Published - https://github.com/aakashns/react-native-dialogs/commit/72080ce7196d63a60022e0596d7c57e528b11702


Get golint tool from golang.org/x/lint/golint

Travis logs:
$ go get github.com/golang/lint/golint
package github.com/golang/lint/golint: code in directory /home/travis/gopath/src/github.com/golang/lint/golint expects import "golang.org/x/lint/golint"
The command "go get github.com/golang/lint/golint" failed and exited with 1 during .

I change CI configuration to use golang.org/x/lint/golint instead of github.com/golang/lint/golint
Would you please review this?
Thank you.

Oh, sorry I fixed this with different p-r 🙇


Add functionality for custom (de)compress instances

Adding these functions make it easier to use custom Decompress and Compress instances. Sometimes these are needed to change the window_bits or zlib_header, without these functions its kinda painful to implement it.
added functions:
bufread

ZlibEncoder::new_with_compress
ZlibDecoder::new_with_decompress

read

ZlibEncoder::new_with_compress
ZlibDecoder::new_with_decompress
ZlibDecoder::new_with_decompress_and_buf

write

ZlibEncoder::new_with_compress
ZlibDecoder::new_with_decompress

Thanks a lot for contributing!
I also see how these new methods would add to an overall more consistent API.
After making adjustments to the doc-strings I'd hope for another untainted pair of eyes for merging though.

Yea this is ny first PR and attempt at making docs :/
Anyway something I should note. (De)Compress both have a function new_with_gzip_header, which feels kinda weird passing to the new_... function of Zlib(De)Encoder...

Thanks for pointing this out!
Yes, maybe a word of caution in the docs might be appropriate. But then again, those who need full control probably know why they need it in the first place.

Currently it doesn't seem possible to set a custom dictionary unless you use the raw Compress/Decompress. But then you don't get access to the private zio module functionality. This PR would enable this

I have given it another review to improve internal consistency, and would think it's ready for merge.
Since the change involves a public API, I'd really love another review, let's see what I can do about that.

@jongiddy Maybe you have thoughts on this PR? I'd appreciate your opinion. Thanks for considering.

I do not think we need new_with_decompress_and_buf. Anyone who needs to control the buffer size should create a BufReader and use bufread::ZlibDecoder directly.
I'm not sure that we need to control the zlib_header. This is an internal flag to allow sharing the implementations for Deflate and Zlib formats. I guess it helps a little if the caller has a similar boolean flag.
That only leaves window_bits. There might be a case for exposing the new_with_window_bits from the mem types, but this should be done consistently in deflate and zlib.
However, given that the mem Compress and Decompress are public already, doing it this way is OK. It provides control over window_bits and provides a way to pass the zlib_header flag directly. My only change would be to remove the new_with_decompress_and_buf method.

Thanks a lot, @jongiddy! I have removed the new_with_decompress_and_buf method as suggested.
This also brings us to the point where we can consider to at least deprecate all the existing new_with_buf() methods while documenting how these should be used instead - they seem redundant after all, but I can imagine how they have been added for convenience. Probably though, it's best to leave these be, while admitting that new_with_decompress is probably the 'expert mode' of constructors, which is used rarely and won't needed such buf related conveniences.
In any case, I think this PR can be merged now.

Oh, never mind, I think after breaking CI in the most sloppy of ways, it became clear why with_buf variants exist. new will always create a BufRead, and controlling the buffer is only possible by having a separate constructor.
It would be possible to let new_with_decompress() take r without wrapping it into BufRead, but that would be quite a departure form the way the API works.
So unless the API is changed to *not * wrap incoming Read impls into BufRead, I think the way it works now (without my change) is the way to go after all.

I am merging the previous state, but please let me know in case I am missing something so we can apply a fix in a follow-up.

Not sure how removing a function that is not used by a test could break CI?
None of the with_buf methods should need to be used as long as the caller does both of:

Wrap a non-BufRead reader in a BufReader; and
Change from the read module to the bufread module for the compress and decompress types.

Oh, never mind, I think after breaking CI in the most sloppy of ways,

Not sure how removing a function that is not used by a test could break CI?

CI breakage was unrelated, or related to me being sloppy.
Related to with_buf(), I meant to say that any change to these would mean deprecating them, yet I thought they are probably there as convenience and thus for a good-enough reason.
However, since the …with_buf() moniker is now causing additional effort as it this PR had to add another method for symmetry, it might be worth rethinking them, while acknowledging that this will affect users in various ways which is why I shied back from a change in the first place.
If you feel it's worth pursuing such a change, I invite you to open a draft PR to sketch it out (which I'd always prefer over an issue which is more abstract).

While the with_buf methods are redundant, I have no desire to deprecate them. My suggestion was to not add the new read::ZlibDecoder::new_with_decompress_and_buf(r, d, vec![0; 1024]) method since the same behaviour can be achieved using bufread::ZlibDecoder::new_with_decompress(BufReader::with_buf(r, vec![0; 1024]), d).
My concern above was that removing this new function, that isn't used anywhere yet, broke CI. But it appears that was not the case.

While the with_buf methods are redundant, I have no desire to deprecate them. My suggestion was to not add the new read::ZlibDecoder::new_with_decompress_and_buf(r, d, vec![0; 1024]) method since the same behaviour can be achieved using bufread::ZlibDecoder::new_with_decompress(BufReader::with_buf(r, vec![0; 1024]), d).

I actually experimented with this which is why I realized it's for the better to keep this PR as is for a consistent API. Would the with_buf methods not exist, it would be clear that new_with_decompress_and_buf would also not exist and for the better. What I wanted to avoid is to have this one 'expert' method that requires the caller to add the BufRead which this implementations currently enforces, causing surprises in performance. This led me to believe that the authors of this type must have believed that having a BufRead is so paramount that there should be no way to even accidentally not have it.
I hope this makes sense.


Azure Powershell task v3.0 inline script has maxlength set to 500

Troubleshooting
Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting
Environment
VSTS

Agent - Hosted

Issue Description
The Azure Powershell task has a text box for inline script set to 500 which is very limiting. Please consider it to 5000 minimum.

Error logs
Cannot type after 500 characters.

@onlyutkarsh
Thanks for raising concern. We have fixed this issue will be deployed within the next 2 weeks.


Bryce/update links for faqs

What does this PR do?
Updates links for integrations FAQs
Motivation
DOCS-3484
Additional Notes
Needs the following PR merged in first:
https://github.com/DataDog/documentation/pull/14437
Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have changelog/ and integration/ labels attached

Hi Bryce, I've added an editorial review card for this.


Something went really wrong! OT-node shutting down

Issue description
Node is shutting down and restarting, happening on couple of nodes, after saving file in /root/ot-node/data/operation_id_cache .[Error: ENOENT: no such file or directory, unlink]
Expected behavior
Otnode should not initiate a restart
Actual behavior
Steps to reproduce the problem

Happens sporadically

Specifications

Node version:v6.0.0-beta.2.2.2
Platform:Ubuntu 20.04
Node wallet:0x9A249D0DD9b8f939b7b5BF139d9198B8DFCBC55f
Node libp2p identity:Qmdki3QeKaQGGowTYz5v59vhz5b6S9R9NouANjuiZc4y7M

Contact details

Email:

Error logs
[2022-09-07 16:27:15] DEBUG: Saving file with name: babe7aec-f6e8-40b6-8d6a-f9f049e4c901 in directory: /root/ot-node/data/operation_id_cache
Something went really wrong! OT-node shutting down... [Error: ENOENT: no such file or directory, unlink '/root/ot-node/data/operation_id_cache/d6792a77-97f1-4d80-94e6-58cc9bb9ab1f'] {
errno: -2,
code: 'ENOENT',
syscall: 'unlink',
path: '/root/ot-node/data/operation_id_cache/d6792a77-97f1-4d80-94e6-58cc9bb9ab1f'
}
[2022-09-07 16:27:15] INFO: Getting assertion for ual: did:otp:0x256736aeb3f19ac6738e9f4d10c9b61da71ceb9f/536156
[2022-09-07 16:27:15] TRACE: publishInitCommand skipped for operationId: 6ccf2b05-8b40-4769-9977-2beb0f7e79e5 with status COMPLETED
[2022-09-07 16:27:15] INFO: Getting assertion for ual: did:otp:0x256736aeb3f19ac6738e9f4d10c9b61da71ceb9f/533109
[2022-09-07 16:27:15] INFO: Assertion integrity validated!
Disclaimer
Please be aware that the issue reported on a public repository allows everyone to see your node logs, node details, and contact details. If you have any sensitive information, feel free to share it by sending an email to<EMAIL_ADDRESS>

Also getting this error
[2022-09-09 11:43:33] DEBUG: Removing file on path: /root/ot-node/data/operation_id_cache/8f86246b-59b4-468b-be3a-a4c0b6d6eb8b
Something went really wrong! OT-node shutting down... [Error: ENOENT: no such file or directory, unlink '/root/ot-node/data/operation_id_cache/8f86246b-59b4-468b-be3a-a4c0b6d6eb8b'] {
errno: -2,
code: 'ENOENT',
syscall: 'unlink',
path: '/root/ot-node/data/operation_id_cache/8f86246b-59b4-468b-be3a-a4c0b6d6eb8b'
}
otnode.service: Main process exited, code=exited, status=1/FAILURE
otnode.service: Failed with result 'exit-code'.
otnode.service: Scheduled restart job, restart counter is at 8203.
Stopped OriginTrail V6 Stage 1 Beta Node.
Started OriginTrail V6 Stage 1 Beta Node.
It ends up in an infinite loop of this

This issue is being closed as inactive due to the date of the last activity on it. If you believe this is still a problem, please create a new issue and confirm that it is reproducible in the current ot-node release version. We are working towards closing open issues that meet specific criteria and ask you to create a new one for those that that are truly bugs in current release. We'll be monitoring those issues so that they are properly managed.
Thank you,
OriginTrail Team


Convert Movie content type to combination of Viewing content type that references a Movie entry

Refactored to use a movie content type that references viewings


[SR-12376] Jenkins fails to clean workspace periodically

Previous ID
SR-12376

Radar
None

Original Reporter
@compnerd

Type
Bug

Additional Detail from JIRA

Votes
0

Component/s

Labels
Bug

Assignee
@shahmishal

Priority
Medium

md5: b5c614a5aae47c2c9a6aea0cb0efebdb

Issue Description:
https://ci-external.swift.org/job/swift-PR-windows/771/
There are many Jenkins bug reports of spurious failures requiring the connection to be restarted as a workaround.  Seems related to JGit.

This should no longer be a problem as we have moved to different system for cleaning up workspace.


CI/CD not working

hello incredible tutorial but I have a problem does not want to compile the app I login with ssh that is a problem? attached img.

now i have this error... I followed your instructions to the letter

this is cloning your repository and putting the file in a repository of mine

https://github.com/NestorDevs/demo-app.git
this repository

could you please re-run the build please. Looks like some transient issue.
I re-ran the build here. And it worked fine. Please find below screenshot

Please re-open it if the issue persists.


About the problem of needing multiple extends

I'm using the cloud storage platform leancloud, but their data builds must extendext AVObject. But I think FastAdapter is a high-performance component, I have a variety of reasons to use it. However, FastAdapter's build method also requires extends from AbstractItem, which makes me encounter obstacles in the development of selection. Who can help solve this problem? Thank you very much.

Need more extends, unable to find a solution.
No related code

Details

[ 3.2.5 ] Used library version
[ 27.1.0 ] Used support library version
[ 4.6 ] Used gradle build tools version
[ 3.1 ] Used tooling / Android Studio version
[ no ] Other used libraries, potential conflicting libraries

@alphater in case you can not use the provided AbstractItem you can always implement the IItem interface.
Make sure you implement the methods accordingly


Can't download lyrics

I added the value of the cookie, and lyrics show up after a search, but those from spotify are empty.
Probably I am missing something because I dont understand very much...
Could you help me?
PS: Sorry for opening a issue for this, but I didnt know how to reach you.

You did everything allright. It was Spotify who stopped providing lyrics to free acounts. I want to add other lyric's sites, but I need time and knowledge to do it :D.

Oh, many thanks for your quick answer. Thanks for your work with the plugin!!

Oh, many thanks for your quick answer. Thanks for your work with the plugin!!

I updated it to use LRCLIB to replace Spotify. Try it if you want.


Simple Maths widget won't forget incorrect commands

To reproduce:
Load some data, e.g. mpdata
Enter an incorrect command in the Simple Maths widget, e.g. mpdata*q
Click 'Run'. Error is displayed as expected.
Now correct the command, e.g. mpdata*4
Click 'Run' Same error is displayed. Can't get it to work although if you run it with a different error it does respond with a different message!

Fixed - the process was not clearing the previous exception before each run so although the process ran using the updated command it then reported the error from the previous one!
For next release

Fixed in 0.9.2 release


Start sevral Containerd servers on one machine at the same time will be a little problem to Ctr client

In this case, i started 4 containers and find that commands will be sent to different server,maybe the latest started one.I don`t know what would be the impact on structure, just want to discuss it.

Ya the ctr client probably need to take the path to the daemon's socket for each one that it wants to communicate with.

totally agree with you. that maybe the best and simplest way to cover it. need to add a new app flag before input command to figure out  which server it is  or give some tips to user there are several starting servers.
how about that ?

I just don't want to make the default use case of one containerd on a system be harder for the user so I think your flag should work well.

Isn't the --address flag for ctr  and --listen one for containerd exactly what you guys were discussing?
In which case, this issue can be closed.

ping @crosbymichael

I'm assuming that the --address and --listen flags from ctr and containerd respectively address the issue since no-one shim'ed in. Feel free to re-open with more details if it's still an issue for you @Issacpeng


Starting service for python script generates 2 different processes

When starting a service installed to run a python script, it works properly, but then it creates 2 separate processes.
I don't know if this is the normal behaviour, but stopping the service only stop one of the two processes, so it keeps alive all the time.
Is this a bug? Any workaround?
Thanks!

forever-service uses forever internally to launch the process. So basically there is a forever monitor process and actual script process. So one nodejs process and one python in this case. If you are seeing two python processes it is most likely that your script is invoking more processes, and forever-service will only try to stop the main processes and not child processes of those. (In most cases). So i would recommend checking your script.

Yes, I was checking it and I think the same, may be a child process of the python script.
So the issue is resolved now (that was fast!). Thanks!


bug: status showing reconnecting while listening

This PR successfully fixes the Issue #1428
No Other File was altered, all tests pass and not a breaking change

@mdmohsin7 please review now and let me know further

@ARYPROGRAMMER could you share with me:
1/ what are your finding so far ? - to ensure you understand about the current issue.
2/ your tests - to confirm the issue has been fixed and nothing else has been broken as far as you know.
happy coding.

1w, no response


[GOBBLIN-613] Using the hadoop tokens fetched by WormholePushJob instead of negotia…

…ting them via Gobblin
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following GOBBLIN-613 issues and references them in the PR title.

Description

[x] Here are some details about my PR, including screenshots (if applicable):
We use the Hadoop tokens provided by the WormholePushJob, instead of relying on Gobblin to negotiate them. Each WormohlePushJob is going to be run with the appropriate headless account and Gobblin does not have access to these keytabs from inside the Azkaban process. So, we need to include WormohlePushJob job type in the job types with automatic token.

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Closing this PR since WormholePushJob is an internal LinkedIn product.


[$100] feat: create linter

We currently use compiler warnings to mark deoptimizations / limitations. What if we created a ESLint plugin for better developing experience?

We currently use compiler warnings to mark deoptimizations / limitations. What if we created a ESLint plugin for better developing experience?

This will be a life-saver!!

/bounty $100

Hi @aidenybai 👋
Would love to take on this, is there a deadline for completion??

Nope, there isn't. Go on ahead and take it @adetoye-dev!

Nope, there isn't. Go on ahead and take it @adetoye-dev!

Thanks Tobi 😊

I will take it

Go ahead @ehizman!

/attempt #394

/attempt
Let me work on this if still not resolved

I think it's safe to close this now @aidenybai. You have merged @Drex72's issue, right?

@aidenybai can I take this up?

Hey this issue is closed ?
I'm interested in working in this. And I do have some question.

We need to publish eslint configs to npm and then use those custom eslint-rules.
I can make skeleton with some predefined rules and then we can add those rules after discussion what react pattern are we targeting which needs user attention.
/attempt #394

/attempt #394

Is this issue Still alive ??? if yes Then I wanna work for it..


GH Management Policy: Add guidance for managing GH organizations

Have a description, have verified links, etc, profile setup with README in landing page

Shipped in #20, closing


add padding to title/logo on navbar

this fixes #338

👍

Thanks for this! You almost had the right fix - but it wasn't quite the correct format. Needed source files, built files and map files :) done here: https://github.com/TryGhost/Casper/commit/11f61782ad582d3f8cc40584bdea45644dae8b85

@JohnONolan okay


Evaluate micro_ros_utilities

https://github.com/micro-ROS/micro_ros_utilities
Is this useful for us?

This is already pulled by microROS project in the build time, allowing the usage of ROS2 native message types in the microcontroller side.


fix(scheduler): remove microsecond component from scheduled_time

Fixes #830
Changes
This ensures that scheduled_time is always lesser than last_scheduled_time_rca if the date, hour, minute and second match exactly. This prevents DeepDrills/Anomaly from being scheduled again if the microsecond component happens to be greater when all other components match.
Tested this on 15 KPIs - none of them displayed the bug.

[ ] Add test cases for this - current time or last scheduled time with non-zero microseconds component

@Samyak2 Can we include this in v0.5.1

Yes, this has been tested thoroughly in an internal instance. Test cases to prevent this issue in the future can be added in a separate PR.

Tests will be added in a separate PR


Height configuration for standing, room, and world target scales

Describe the problem
Currently, when using MRTK, the MRTK camera and Playspace are both positioned at the origin upon initialization, with adjustments to the camera position offsetting it from the parent playspace object. When trying to set the camera to start at a relative "height" compared to the playspace, the user ends up higher than they would expected based on just the editor window.
In addition, without setting this "height", teleportation across surfaces ends up teleporting users to floor level, as the default "height" of the user is set to be zero.
What we need to do is to be able to easily set a relative "height" for the user for standing, room and world target scale experiences so that can provide a better experience with the teleportation move system and other systems that might need to take the users's height into account

@RogPodge moving this out to MRTK.Next. If we have a branch with this we can reconsider

@RogPodge moving this out to MRTK.Next. If we have a branch with this we can reconsider

That's fine, at the moment it's looking to be part of the core usability improvements we're making


Use p2p for fetching blocks from bitcoind instead of rpc

Fetching raw blocks via RPC is much slower than via p2p if the node is listening.
The RPC interface requires parsing the json request, fetching the block from disk and deserializing it into CBlock, serializing it into hex, then putting it into json and sending over the wire. The client also has to deserialize the json response and then the block from hex.
The RPC interface also requires authentication and is rate limited. rpcworkqueue and rpcthreads should be configured for more reliable indexing.
P2P requires listen=1 to be set (it is default but some configurations set it to listen=0 by default, like when setting connect), and requires a handshake before serving. After the handshake, blocks are fetched directly from disk without deserializing and sent across the wire in binary form. It requires no authentication and will not fall over from exceeding work queue.
See https://github.com/romanz/electrs/blob/master/src/p2p.rs.

Using REST is also a more suitable application for this than RPC. It requires no authentication and can transmit blocks in binary form. It also does not have to parse a json request. It does requires rest=1 in config though. However, it still deserializes a block to CBlock internally before sending. Ideally https://github.com/bitcoin/bitcoin/pull/26415 would be merged and this would be by far the fastest way to fetch blocks.
See https://github.com/andrewtoth/rust-bitcoincore-rest.


POC of code level optimization for Hive

This is currently WIP

@varunsaxena As mentioned in the description , this is still work in progress . Please don't review it now , as there will be considerable changes in the PR

@varunsaxena Can u please review it

@mkumar1984 Please review the same


question: how to specify helm chart version

In 0.5.4, the version field was dropped. What's the best way to specify the helm chart version?

Hey. We now define version directly in the config file.
here is the example: https://docs.cosmology.zone/starship/config


sharing same token among multiple apps

I have two android apps that need to share the same access token. I save token in App A, and try to access that token from App B. While it works in debug mode, I can't read accounts that saved by other app when it's built in release mode.

Is there an extra step to achieve the goal?
If I set the same accountType, the app that's installed secondly doesn't open and just shows blank page.

@ghostman2013 Would you have any idea how to access App A's token from App B?

@yakppal   Hi. Did you find a solution regarding this issue?


First attempt Unit Testing

We did Unit Test on methods (info, debug, annoy, warn, error, fatal, and get_logging_path).

Recommend adding the description of the Pull Request in comment above to the PR description so it's easier to find.  Also, the description needs to give a high level summary of what changed (which you did), but also why
Example:
Adding unit tests around the existing slogpy public logging methods focused on validating output to the log file and console.  Tests are intended to support upcoming changes to prevent regressions.

Hey @zhenhuaizeng I think we discussed having the tests also sharing the same input. In the case of the tests above, all of them are using a different input, which means that you are potentially testing string combinations as well. This matters because things like escape characters or symbols (e.g. "!") can behave differently when used in strings. I believe Alex from the team brought this up.
I would suggest you test that separately as well in a different pull request.
Additionally, some documentation, what was the task/bug that you are working on that is related to the pull request? (e.g link to the basecamp task, or the documentation that Sagal is creating)

@zhenhuaizeng see my comment above about filling out the description of the PR https://github.com/slogpy/slogpy/pull/3#issuecomment-2081531205

Adding unit tests around the existing slogpy public logging methods focused on validating output to the log file and console. Giving 'happy path' for each of the methods on the public interface(info,debug,annoy,warn,error,fatal) to ensure they log the expected message.

We removed the "test_version" method from the test_slogpy_console.py file because it was not testing console output.

Restored the test_slogpy_file to the VS code and uploaded the test_slogpy_file to Git Hub as our second task, as described on Basecamp.

Closing this as it is now very old and obsolete.


feat: migrate xcall to seperate repo

Description:
Commit Message
feat: migrate xcall to seperate repo

see the guidelines for commit messages.
Changelog Entry
version: <log entry>

Checklist:

[ ] I have performed a self-review of my own code
[ ] I have documented my code in accordance with the documentation guidelines
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have run the unit tests
[ ] I only have one commit (if not, squash them into one commit).
[ ] I have a descriptive commit message that adheres to the commit message guidelines

Please review the CONTRIBUTING.md file for detailed contributing guidelines.

Codecov Report

Merging #522 (0e3abb8) into main (4dd5480) will decrease coverage by 0.52%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main     #522      +/-   ##
============================================
- Coverage     67.02%   66.50%   -0.52%
+ Complexity      580      404     -176
============================================
  Files           195      183      -12
  Lines         17738    16989     -749
  Branches        371      292      -79
============================================
- Hits          11889    11299     -590
+ Misses         5656     5534     -122
+ Partials        193      156      -37

Flag
Coverage Δ

java
80.00% <100.00%> (+0.33%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

contracts/cosmwasm-vm/cw-common/src/lib.rs
97.56% <ø> (ø)

.../cosmwasm-vm/cw-common/src/xcall_connection_msg.rs
0.00% <ø> (ø)

.../main/java/ibc/xcall/connection/IBCConnection.java
89.13% <100.00%> (+0.95%)
:arrow_up:


On search title is loosing styling

On search title losing formatting, it could be i'm doing something wrong or this is an actual issue.
I'm attaching sample code for quick test
Screenshot:

Video:
Recording 2022-10-10 18-01-33 from Snipclip.webm
Quick sample code:
package main

import (
	"encoding/json"
	"fmt"
	"os"

	"github.com/charmbracelet/bubbles/list"
	tea "github.com/charmbracelet/bubbletea"
	"github.com/charmbracelet/lipgloss"
)

var docStyle = lipgloss.NewStyle().Margin(1, 2)
var special = lipgloss.AdaptiveColor{Light: "#43BF6D", Dark: "#73F59F"}
var strike = lipgloss.NewStyle().
	Strikethrough(true).
	Foreground(lipgloss.AdaptiveColor{Light: "#969B86", Dark: "#696969"})

var checkMark = lipgloss.NewStyle().SetString("✓").
	Foreground(special).
	PaddingRight(1).
	String()
var crossMark = lipgloss.NewStyle().SetString("x").
	Foreground(lipgloss.AdaptiveColor{Light: "#f44336", Dark: "#cc0000"}).
	PaddingRight(1).
	String()

type Response struct {
	Resources []resource `json:"resources"`
}

type resource struct {
	Name   string `json:"name"`
	Status string `json:"status"`
}

func (i resource) Title() string {
	name := i.Name
	if i.Status == "Running" {
		return fmt.Sprintf("%s%s", checkMark, name)
	}
	return crossMark + strike.Render(name)
}

func (i resource) Description() string {
	return ""
}

func (i resource) FilterValue() string { return i.Name }

type model struct {
	list list.Model
}

func (m model) Init() tea.Cmd {
	return nil
}

func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
	switch msg := msg.(type) {
	case tea.KeyMsg:
		if msg.String() == "ctrl+c" {
			return m, tea.Quit
		}
	case tea.WindowSizeMsg:
		h, v := docStyle.GetFrameSize()
		m.list.SetSize(msg.Width-h, msg.Height-v)
	}

	var cmd tea.Cmd
	m.list, cmd = m.list.Update(msg)
	return m, cmd
}

func (m model) View() string {
	return docStyle.Render(m.list.View())
}

func main() {
	var response Response
	err := json.Unmarshal([]byte("{\"resources\":[{\"name\":\"adam-test-1\",\"status\":\"Running\"},{\"name\":\"test-2\",\"status\":\"Running\"},{\"name\":\"test-3\",\"status\":\"Running\"}]}"), &response)
	if err != nil {
		panic(err)
	}
	items := []list.Item{}
	for _, v := range response.Resources {
		items = append(items, v)
	}
	m := model{list: list.New(items, list.NewDefaultDelegate(), 0, 0)}
	m.list.Title = "Testing"
	p := tea.NewProgram(m, tea.WithAltScreen())
	if err := p.Start(); err != nil {
		fmt.Println("Error running program:", err)
		os.Exit(1)
	}
}

Setting styles inside list items isn't currently supported and I'm not sure we should support that. In such a case you probably want to write your own item delegate to influence how items gets rendered. You can see an example of how that's done in the list-fancy example: https://github.com/charmbracelet/bubbletea/tree/master/examples/list-fancy

Style inside of list is working fine when it's rendered first, it only breaks when you start searching. It's not even breaking on pagger/pagiation.

@abhimanyu003 is it breaking when you search if you set styling through the item delegate as muesli recommended? That is the way we suggest customizing list item styling. If there's an issue with it, please let us know

Going to move this to a discussion, if there's a bug or anything actionable that needs to take place after, we can open an issue to address it


add passThrough option

In case you need to bypass the mock and issue real http request

This change is great, but can you please write a test for the new case similar to https://github.com/omer72/angular-multimocks/blob/master/app/src/js/multimocks.spec.js#L132

Is this something your able to add a test for? would be a nice addition.


error in saving model with deepspeed

Thanks for your great work. I can successfully train model by your proposed code in A100 without any error. However, training in V100 with deepspeed, I meet  "Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data" when I save checkpoint to disk. My deepspeed config is
compute_environment: LOCAL_MACHINE deepspeed_config:  gradient_accumulation_steps: 1  gradient_clipping: 1.0  offload_optimizer_device: none  offload_param_device: none  zero3_init_flag: true  zero_stage: 2 distributed_type: DEEPSPEED fsdp_config: {} machine_rank: 0 main_process_ip: null main_process_port: null main_training_function: main mixed_precision: fp16 num_machines: 1 num_processes: 8 use_cpu: false
can you give me some suggestion, thanks.

same problem

from pasting your comment into chatgpt v4
It's great to hear that you were able to successfully train your model using an A100 GPU. However, encountering an error with NCCL operations while training on a V100 GPU with DeepSpeed is indeed a concern that needs to be addressed. The error message you're encountering, "Some NCCL operations have failed or timed out," typically indicates issues related to GPU communication in distributed training setups. Here are some suggestions to troubleshoot and potentially resolve this issue:

1. **NCCL Debugging**: Enable NCCL debugging to get more detailed logs. This can be done by setting the environment variable `NCCL_DEBUG=INFO` or `NCCL_DEBUG=WARN`. This will provide more insights into what might be going wrong with the NCCL operations.

2. **Check GPU Compatibility**: Ensure that the V100 GPUs are compatible with the version of NCCL and CUDA you are using. Sometimes, specific versions of these libraries have compatibility issues with certain GPU architectures.

3. **DeepSpeed Configuration**: Review your DeepSpeed configuration. You are using ZeRO Stage 2, which is quite memory-efficient but can be demanding in terms of inter-GPU communication. You might want to experiment with ZeRO Stage 1 or even disabling ZeRO to see if the problem persists.

4. **Reduce Model Size or Batch Size**: Temporarily reduce the size of your model or the batch size to check if it's a memory issue. Over-subscription of GPU memory can sometimes lead to obscure errors.

5. **Update Libraries**: Ensure that DeepSpeed, PyTorch, CUDA, and NCCL are all updated to their latest versions. Sometimes, bugs in these libraries can cause such issues, and updates often contain fixes.

6. **Network Communication**: If you are training across multiple machines or using a multi-GPU setup, there could be network-related issues. Check the network bandwidth and stability.

7. **File System Load**: The error occurring at the checkpoint saving phase could indicate a file system bottleneck. Ensure that the file system can handle the I/O load, especially if you're saving large checkpoints.

8. **Hardware Check**: Run some diagnostics on your V100 GPUs to ensure they are functioning correctly. Hardware issues, though rare, can lead to such errors.

9. **Experiment with `gradient_accumulation_steps`**: Sometimes, changing the `gradient_accumulation_steps` can impact the way memory is handled. Try increasing it to see if it affects the stability of your training.

10. **Consult DeepSpeed Documentation/Community**: Check the DeepSpeed documentation for any similar issues or considerations. Also, reaching out to the DeepSpeed community (like GitHub issues or forums) with your specific error can be helpful.

Remember, distributed training, especially with advanced optimizations like ZeRO, can be quite complex, and issues might stem from a combination of factors including software configuration, network setup, and hardware capabilities.

I meet the same problem. Have you solved it?

The deepspeed needs to save model in all processes, not only the main process. So just remove the accelerator.is_main_process in the saving part should work. Otherwise, other processes will wait for saving models and will cause NCCL timeout error after 30 minutes.


Upload cookbook to server stopped working

My release stopped working, cause the task is failing in berks install.
Last time it worked the task was in version 1.1.6

2017-12-20T12:24:39.7833653Z ##[section]Starting: Upload cookbook to Chef Server
2017-12-20T12:24:39.8099541Z ==============================================================================
2017-12-20T12:24:39.8205094Z Task         : Upload cookbook to Chef Server (Chef)
2017-12-20T12:24:39.8264285Z Description  : Upload a cookbook to Chef Server including dependencies
2017-12-20T12:24:39.8323053Z Version      : 1.2.63
2017-12-20T12:24:39.8383323Z Author       : Chef
2017-12-20T12:24:39.8442761Z Help         : More information
2017-12-20T12:24:39.8504261Z ==============================================================================
2017-12-20T12:24:40.6013028Z 98693444-e22b-415d-bd50-85bf94580813 exists true
2017-12-20T12:24:40.7038312Z CD to cookbook directory: /opt/vsts/work/1/s/cookbooks/chef-bp
2017-12-20T12:24:40.7250145Z [command]/opt/chefdk/bin/berks install
2017-12-20T12:24:40.7496635Z ##[error]/opt/chefdk/bin/berks failed. spawn /opt/chefdk/bin/berks ENOENT
2017-12-20T12:24:40.7860669Z [command]/opt/chefdk/bin/berks upload -c /tmp/berks.config.json
2017-12-20T12:24:40.8044326Z ##[error]/opt/chefdk/bin/berks failed. spawn /opt/chefdk/bin/berks ENOENT
2017-12-20T12:24:40.8169623Z Removing Private key: /tmp/vsts-task.pem
2017-12-20T12:24:40.8479082Z ##[section]Finishing: Upload cookbook to Chef Server

Hello @joricteixeira
That is not good. Please could you tell me what version of ChefDK you are using in your agent?

Hi there @russellseymour , thanks for the fast response
I'm currently using Linux Hosted Agent. I'm not quite sure how to retrieve that version information.
I've tried to run with system.debug=true and the only log regarding the chefDK was the following:

2017-12-21T16:49:07.6457294Z ##[debug]chefDKChannel=null
2017-12-21T16:49:07.6530875Z ##[debug]chefDKVersion=null
2017-12-21T16:49:07.6604977Z ##[debug]chefDKForceInstall=null

Hey @russellseymour, after your question I've bumped into a task "Install ChefDK".
After I added this task before the upload task itself, it started working again.
I guess the lesson learned is: the Upload task used to work despite "manual" pre-installation of the ChefDK. It doesn't anymore.
Thank you so much for the insight!


Miri evaluation errors show up with error code E0080

When running MIRI on a Rust program, I was a bit puzzled with the output error message with the error code E0080 as the code seemed irrelevant to the error that MIRI catched.
error[E0080]: Miri evaluation error: trying to reborrow for SharedReadWrite, but parent tag <1368> does not have an appropriate item in the borrow stack
After running MIRI on another Rust program did I begin to think that maybe all MIRI evaluation errors are printed out with the same rustc error code E0080.
(Assuming all MIRI evaluation errors are printed out with code E0080)
To prevent unnecessary confusion, maybe it would be nice to either

create a new Rustc error code dedicated for MIRI evaluation errors OR
add an additional explanation to the rustc --explain E0080 message,  mentioning that E0080 can also be relevant to MIRI evaluation errors.

all MIRI evaluation errors are printed out with the same rustc error code E0080.

Lol, I had never even noticed this.^^
@oli-obk is there something we can do about this? Ideally we'd not have an error code, or have an error code dedicated to Miri.

We can remove the error code entirely. There's no point to miri having a specific error code.

The problem is that we are currently calling this method:
pub fn struct_error<'tcx>(tcx: TyCtxtAt<'tcx>, msg: &str) -> DiagnosticBuilder<'tcx> {
    struct_span_err!(tcx.sess, tcx.span, E0080, "{}", msg)
}

It's a one-liner, so we could inline it to change the error code, but what would we do do not have an error code?

https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html#method.struct_span_err

Macro struct_span_err! makes a call to fn struct_span_err_with_code.
Under the miri/src directory, the only file that contains a call to the fn struct_error is miri/src/eval.rs.


In default, sync commit is true. Then commit callback won't be invoked which results ConcurrentMessageListenerContainerTests.testManualCommitExisting testing failed.

@addozhang Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

Wrong commit msg


[Task] refactor provider client sdks

Description
Now there are some duplicate codes in the client SDK. This issue aim to solve the problem.
There are some useful information in https://github.com/apache/shenyu/pull/3912 and https://github.com/apache/shenyu/pull/3931 .
Task List

[ ] refactor shenyu-client-alibaba-dubbo;

[ ] refactor shenyu-client-grpc;

[ ] refactor shenyu-client-springcloud;

[ ] refactor shenyu-client-motan;

[ ] refactor shenyu-client-sofa;

[ ] refactor shenyu-client-tars;

[ ] refactor shenyu-client-websocket;

pls asign task 1 to me. thanks

pls asign task 2 to me. thanks

pls assign task 3 to me. Thx

Please assign task 7 to me, thanks!

pls asign task 5 to me. Thx

Please assign task 6 to me, thanks!

@dragon-zhang Hi, I found the appName assigned different value when refactoring  shenyu-client-springcloud:
In the original constructor of SpringCloudClientEventListener:

But in the constructor of AbstractContextRefreshedEventListener:

Should I keep the original or use the superclass assignment?

@dragon-zhang Hi, I found the appName assigned different value when refactoring shenyu-client-springcloud: In the original constructor of SpringCloudClientEventListener:
But in the constructor of AbstractContextRefreshedEventListener:
Should I keep the original or use the superclass assignment?

keep the original, and you need to override getAppName.

Good job, guys !


fix for navbar wrapping text issue

Fix for #138

@Shubhangam333 Thank you so much for finding several places to improve in the website and fixing them.  Awesome job!  :100:


Wrong date

Hi,
this code
console.log(hebrewDate(new Date(2021, 8, 18)))

gives me
{
    "year": 5782,
    "month": 1,
    "date": 12,
    "month_name": "Tishri"
}

Instead of 10 Ellul.
What's wrong here?

months are zero indexed, Date(2021, 7, 18) will give you correct results


az ad sp create-for-rbac fails even after deleting the created sp

Description
It seems like az ad sp create-for-rbac is creating a service principal only, so after running az ad sp delete, deleting the sp leaves the application (which makes sense).
The problem is when deleting a sp created with create-for-rbac, then trying to run az ad sp create-for-rbac, the command fails unless you delete the app.
Seems this command should be az ad app create-for-rbac or take into account the app may still exist and simply add another sp to the existing app.

Actually deleting  the app is the right way. In the output of "-h", there is a section
Revoke the service principal when done with it.
        az ad app delete --id <name>

i have no push back that the experience is not optimal.
I don't think you can create more than 1 sp for an app, but we can at least output a clear actionable error to delete the app, or reuse it if we can  put in new creds.

I will expose a command of az ad sp revoke-for-rbac to clean up related artifacts across, including the role assignments

Why don't we just make the sp delete command default to deleting the app definition unless you specify a --keep-app flag or something like that?

sorry, i didn't update the comment in time. Yep, I chatted with @devigned yesterday, I will let both ad sp/app delete to clean up the assignment, and related sp object as well


Add emoji stream to Powered By

Hi @muan, thanks so much for this lib 😄👍
This PR adds emoji-stream under the "Powered by emojilib" section in the README.
Related to #45.

:v: Cool project! Thanks!


docs: draft on improving commit context

What this PR does
RFC draft on improving commit context
Which issue(s) this PR resolves
Ref: #125
Anything else

This change is

Codecov Report

Merging #149 (dc5f15e) into main (1340bba) will increase coverage by 0.07%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #149      +/-   ##
==========================================
+ Coverage   62.86%   62.93%   +0.07%
==========================================
  Files         126      126
  Lines       17049    17049
==========================================
+ Hits        10718    10730      +12
+ Misses       5781     5773       -8
+ Partials      550      546       -4

Impacted Files
Coverage Δ

internal/storagenode/logstream/executor.go
80.16% <0.00%> (-0.64%)
:arrow_down:

internal/metarepos/storage.go
81.37% <0.00%> (+0.23%)
:arrow_up:

internal/metarepos/report_collector.go
78.67% <0.00%> (+1.07%)
:arrow_up:

...metarepos/dummy_storagenode_client_factory_impl.go
63.88% <0.00%> (+1.38%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@hungryjang
I modified a hyperlink after your approval.
https://github.com/ijsong/varlog/blob/84047c5125ccc1f7f493cf352fa3f8c1399d485f/docs/RFCs/20220915_commit_context.md?plain=1#L245


Create Body prefab and add to all demo scenes

Overview
All of the scenes had the same hierarchy of Body/Head for the camera. This now refactors that out into a prefab and adds the controller visualizer script to the prefab (and thus all scenes).

Closed until some other PRs get in.

We shouldn't need prefabs anyway.


Doesn't work/sync across devices

If possible include the option to sync the data to other devices or have an option to connect to gmail and send out an email on a weekly basis.

E-mail thing is definitely crossing the line, plus it would require additional permissions (i.e. permission to query remote servers) that I'm not comfortable having users agree to since I don't want people to make a habit of naivety in granting apps/extensions such potentially dangerous abilities.
As for syncing across devices, it's something that could be added once basic security is introduced. AFAIK credentials are just stored in plaintext currently, not something I want users sending to Google willy-nilly (it's also notably unsafe just running locally in its present state... maybe Chrome has built-in crypto encapsulating local storage for sites/extensions, but I'd be impressed).


Investigate issues around auto importing

"auto-import doesn't seem to work - I don't see any errors, it just doesn't seem to run" - Vlad from Discord
May want to look into stability solutions for SmartBlocks, then abstract framework into roamjs-components for reuse here

Any updates on this? 👀

Moved it up the RoamJS Backlog

look into backend API, scheduled job

#17  should fix the default auto-import issue.
But another issue will still remain for multiple imports: https://github.com/samepage-network/otter-samepage/issues/4


Runtime Error: OOM issue in segformer efficient selfattention pipeline

Describe the bug
Facing OOM issue from permute in the segformer efficient selfattention pipeline for certain shapes. I have taken a snippet of the sub_module to reproduce the issue.
To Reproduce
Steps to reproduce the behavior:

Checkout to branch, punith/segformer_issues

Run command, pytest models/experimental/functional_segformer/segformer_unit_test_1.py

Expected behavior
The test should pass without OOM issue
Screenshots
E       RuntimeError: TT_THROW @ ../tt_metal/impl/allocator/allocator.cpp:141: tt::exception
E       info:
E       Out of Memory: Not enough space to allocate 33554432 B L1 buffer across 64 banks, where each bank needs to store 524288 B

Please complete the following environment information:

OS:  Ubuntu 20.04
Device: WH-n150

Additional context
If we remove the tensor from device and then keep on device, we are not facing that issue.
Test passes without any issue if we uncomment  line 22 & 26 in file segformer_unit_test_1.py
CC: @saichandax

@dvartaniansTT @punithsekar initially I thought it was a sharding issue but it looks like the op might just be using too much L1. its currently NOT sharded and trying to allocate all of it to a single L1. Is that intententional?

Hey @bbradelTT , I think @dvartaniansTT @punithsekar intended to use sharded L1.
At the ttnn API level they used the ttnn.L1_MEMORY_CONFIG which is a interleaved L1 mem config.
I am not sure if under the hood linear uses that to shard (because they also gave it a core grid separately).
Could we clean up the API to use the sharded memory config if the intention is to shard? And then fix this OOM issue assuming it's related to the buffer not being sharded properly.

@tarafdarTT if the user tells us to use a specific memory config for linear, we are not going to go behind their back and use a different one.
The user needs to specify the memory config that they want or not specify one and let linear base the decision on the inputs, although in that case, the inputs need to be set up properly to have the right memory config.
core_grid is just overwriting the device core grid and can be used to restrict which cores are used. Because of backwards compatibility it also changes the algorithm for automatically choosing a program config.

@tarafdarTT if the user tells us to use a specific memory config for linear, we are not going to go behind their back and use a different one.
The user needs to specify the memory config that they want or not specify one and let linear base the decision on the inputs, although in that case, the inputs need to be set up properly to have the right memory config.
core_grid is just overwriting the device core grid and can be used to restrict which cores are used. Because of backwards compatibility it also changes the algorithm for automatically choosing a program config.

I see so the issue is the incorrect memory config that is allocating too large of an L1 buffer.  Sorry about that I was confused because of the interleaved memory config plus core grid.
@dvartaniansTT @punithsekar please update your memory config to a sharded memory config:
refer to tests/ttnn/unit_tests/operations/test_core.py how to do that.

Also, I don't think it matters if the tensor is sharded or not.
33554432 B L1 buffer across 64 banks, where each bank needs to store 524288

indicates that it just won't fit in 64 cores.
You probably need to store the tensor(s) in DRAM.

Also, I don't think it matters if the tensor is sharded or not.
33554432 B L1 buffer across 64 banks, where each bank needs to store 524288

indicates that it just won't fit in 64 cores.
You probably need to store the tensor(s) in DRAM.

@punithsekar @dvartaniansTT

@punithsekar @dvartaniansTT
Why is the memory usage so large?
 You might be able to just move the tensor over to DRAM before permute.
I think that's what the commented out lines do.